Jul 31 23:44:12.923160 ip-10-0-140-3 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 31 23:44:12.923172 ip-10-0-140-3 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 31 23:44:12.923179 ip-10-0-140-3 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 31 23:44:12.923411 ip-10-0-140-3 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 31 23:44:23.172962 ip-10-0-140-3 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 31 23:44:23.172979 ip-10-0-140-3 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b143242ad79b49f9b427937b97d1ec41 --
Jul 31 23:45:59.457384 ip-10-0-140-3 systemd[1]: Starting Kubernetes Kubelet...
Jul 31 23:45:59.908763 ip-10-0-140-3 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:45:59.908763 ip-10-0-140-3 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 31 23:45:59.908763 ip-10-0-140-3 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:45:59.908763 ip-10-0-140-3 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 31 23:45:59.908763 ip-10-0-140-3 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:45:59.909944 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.909847    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 31 23:45:59.914924 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914904    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:45:59.914924 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914922    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:45:59.914924 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914926    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:45:59.914924 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914929    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914932    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914937    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914942    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914945    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914948    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914952    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914955    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914958    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914962    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914964    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914967    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914969    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914978    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914981    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914984    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914986    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914989    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914992    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914994    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:45:59.915073 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914997    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.914999    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915003    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915006    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915009    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915013    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915016    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915018    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915021    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915023    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915026    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915029    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915032    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915035    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915052    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915056    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915059    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915062    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915064    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:45:59.915576 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915067    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915069    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915072    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915075    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915078    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915080    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915083    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915086    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915089    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915092    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915095    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915097    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915100    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915102    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915105    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915108    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915112    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915115    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915118    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915121    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:45:59.916056 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915123    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915126    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915128    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915131    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915134    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915137    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915139    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915142    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915145    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915148    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915150    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915153    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915157    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915160    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915163    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915165    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915168    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915171    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915173    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:45:59.916536 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915175    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915179    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915181    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915184    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915187    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915189    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915192    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915194    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915197    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915199    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915202    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915204    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915206    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915209    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915211    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915214    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915217    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915220    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915972    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915978    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:45:59.916993 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915983    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915987    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915989    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915993    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915995    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.915998    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916001    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916004    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916006    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916009    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916012    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916014    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916017    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916019    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916023    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916026    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916028    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916031    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916033    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916036    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:45:59.917500 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916054    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916058    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916063    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916066    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916069    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916072    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916076    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916080    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916084    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916087    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916090    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916092    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916096    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916099    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916101    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916104    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916107    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916109    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916112    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:45:59.918345 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916114    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916117    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916119    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916122    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916125    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916128    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916130    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916133    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916136    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916139    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916142    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916144    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916147    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916150    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916152    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916155    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916158    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916161    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916164    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916167    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:45:59.919096 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916170    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916173    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916175    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916178    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916182    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916185    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916188    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916190    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916193    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916196    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916198    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916201    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916203    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916206    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916208    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916211    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916213    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916216    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916218    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916221    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:45:59.919645 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916223    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916226    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916229    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916231    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916234    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916237    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916240    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916242    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916245    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916247    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916249    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916252    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916256    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916260    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916263    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916266    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916269    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916272    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916678    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:45:59.920163 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916683    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916686    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916689    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916693    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916696    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916699    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916701    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916705    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916708    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916710    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916713    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916716    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916719    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916721    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916724    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916728    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916732    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916735    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916738    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916740    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:45:59.920624 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916743    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916746    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916749    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916751    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916754    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916757    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916760    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916762    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916765    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916768    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916770    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916773    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916776    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916779    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916781    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916784    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916786    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916789    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916791    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916794    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:45:59.921148 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916796    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916799    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916802    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916804    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916807    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916809    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916811    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916814    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916817    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916820    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916822    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916825    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916827    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916830    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916832    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916835    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916837    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916840    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916842    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916845    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:45:59.921627 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916847    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916853    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916855    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916858    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916860    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916863    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916865    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916868    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916870    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916873    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916875    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916878    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916880    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916882    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916885    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916888    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916890    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916892    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916895    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916897    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:45:59.922173 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916900    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916904    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916907    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916909    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916911    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916914    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916917    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916919    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916922    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916925    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916927    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916930    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916932    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916936    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916940    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916944    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916947    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.916949    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917023    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917031    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 31 23:45:59.922753 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917036    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917055    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917060    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917063    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917068    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917072    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917075    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917078    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917081    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917085    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917088    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917091    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917095    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917099    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917102    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917105    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917109    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917112    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917115    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917118    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917121    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917125    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917128    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 31 23:45:59.923336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917131    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917134    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917140    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917143    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917146    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917149    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917153    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917157    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917160    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917163    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917166    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917169    2578 flags.go:64] FLAG: --enable-server="true"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917172    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917176    2578 flags.go:64] FLAG: --event-burst="100"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917179    2578 flags.go:64] FLAG: --event-qps="50"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917181    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917184    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917187    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917191    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917194    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917197    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917201    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917204    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917207    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917211    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 31 23:45:59.923956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917214    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917217    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917219    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917222    2578 flags.go:64] FLAG: --feature-gates=""
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917226    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917229    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917232    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917235    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917239    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917243    2578 flags.go:64] FLAG: --help="false"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917246    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-140-3.ec2.internal"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917249    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917252    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917255    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917258    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917262    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917265    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917268    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917271    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917274    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917277    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917280    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917283    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917286    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 31 23:45:59.924659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917289    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917292    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917295    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917298    2578 flags.go:64] FLAG: --lock-file=""
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917301    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917303    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917306    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917312    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917315    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917318    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917321    2578 flags.go:64] FLAG: --logging-format="text"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917324    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917327    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917330    2578 flags.go:64] FLAG: --manifest-url=""
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917333    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917337    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917340    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917345    2578 flags.go:64] FLAG: --max-pods="110"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917348    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917351    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917354    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917357    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917360    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917363    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917366    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 31 23:45:59.925279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917373    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917376    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917380    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917383    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917386    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917391    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917394    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917397    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917400    2578 flags.go:64] FLAG: --port="10250"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917403    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917406    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03f415a79bb9bded5"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917409    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917413    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917415    2578 flags.go:64] FLAG: --register-node="true"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917418    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917426    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917429    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917432    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917435    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917439    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917442    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917445    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917448    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917451    2578 flags.go:64] FLAG: --runonce="false"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917455    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 31 23:45:59.925898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917459    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917462    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917464    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917467    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917471    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917473    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917476    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917479    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917482    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917485    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917488    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917491    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917494    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917497    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917502    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917506    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917509    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917513    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917516    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917519    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917522    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917525    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917528    2578 flags.go:64] FLAG: --v="2"
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917533    2578 flags.go:64] FLAG: --version="false"
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917537    2578 flags.go:64] FLAG: --vmodule=""
Jul 31 23:45:59.926618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917541    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.917544    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918710    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918727    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918732    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918737    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918743    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918748    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918752    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918762    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918767    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918771    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918776    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918781    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918785    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918790    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918794    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918799    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918804    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918808    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:45:59.927243 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918812    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918816    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918828    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918832    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918836    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918843    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918850    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918854    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918859    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918863    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918867    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918873    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918878    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918882    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918887    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918896    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918900    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918905    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918911    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918915    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:45:59.927736 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918920    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918924    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918928    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918932    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918937    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918941    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918946    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918955    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918959    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918963    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918968    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918972    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918977    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918981    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918987    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918991    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918995    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.918999    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919003    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919012    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:45:59.928235 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919016    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919020    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919025    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919030    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919036    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919058    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919063    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919067    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919073    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919079    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919086    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919097    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919103    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919107    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919112    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919117    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919121    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919125    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919130    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:45:59.928715 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919134    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919138    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919143    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919147    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919151    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919160    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919164    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919170    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919174    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919179    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919183    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919189    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919193    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919198    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919202    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919206    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919211    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919219    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919225    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919230    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:45:59.929199 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919268    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:45:59.929692 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.919719    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:45:59.929692 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.920592    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:45:59.929692 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.928093    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 31 23:45:59.929692 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.928110    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 31 23:45:59.929692 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.928130    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 31 23:45:59.929692 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.928135    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 31 23:45:59.929692 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928201    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:45:59.929692 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928207    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:45:59.929692 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928211    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:45:59.929692 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928214    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:45:59.929692 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928217    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:45:59.929692 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928219    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:45:59.929692 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928221    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:45:59.929692 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928224    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:45:59.929692 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928227    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928230    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928233    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928236    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928238    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928241    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928244    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928246    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928249    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928252    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928254    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928257    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928259    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928262    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928265    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928267    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928271    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928273    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928276    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928278    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:45:59.930100 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928280    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928284    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928287    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928291    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928294    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928297    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928300    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928303    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928305    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928308    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928310    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928313    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928315    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928318    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928320    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928323    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928326    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928328    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928331    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928333    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:45:59.930589 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928336    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928338    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928341    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928344    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928346    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928349    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928351    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928354    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928356    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928359    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928361    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928364    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928367    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928370    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928373    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928376    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928378    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928381    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928383    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928386    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:45:59.931184 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928388    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928391    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928393    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928396    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928398    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928401    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928403    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928406    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928408    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928411    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928414    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928416    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928419    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928423    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928426    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928429    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928433    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928436    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928439    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928442    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:45:59.931666 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928445    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:45:59.932160 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928448    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:45:59.932160 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928450    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:45:59.932160 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928453    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:45:59.932160 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928455    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:45:59.932160 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928458    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:45:59.932160 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928461    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:45:59.932160 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928464    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:45:59.932160 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928468    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:45:59.932160 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928471    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:45:59.932160 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928474    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:45:59.932160 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.928479    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:45:59.932160 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928594    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:45:59.932160 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928598    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:45:59.932160 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928601    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928604    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928607    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928610    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928612    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928615    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928617    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928620    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928622    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928625    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928627    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928630    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928632    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928635    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928637    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928639    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928642    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928645    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928647    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:45:59.932495 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928651    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928654    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928657    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928660    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928663    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928666    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928669    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928671    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928674    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928676    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928678    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928681    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928683    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928686    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928688    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928691    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928693    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928695    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928698    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928701    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:45:59.932950 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928703    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928706    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928710    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928713    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928716    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928719    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928722    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928725    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928728    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928731    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928734    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928736    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928739    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928742    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928744    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928747    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928750    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928752    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928755    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:45:59.933442 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928757    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928760    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928762    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928765    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928768    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928770    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928772    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928775    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928777    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928780    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928783    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928786    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928789    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928792    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928794    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928797    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928800    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928803    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928805    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928808    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:45:59.933882 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928810    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:45:59.934392 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928812    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:45:59.934392 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928815    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:45:59.934392 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928817    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:45:59.934392 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928820    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:45:59.934392 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928822    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:45:59.934392 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928825    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:45:59.934392 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928827    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:45:59.934392 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928830    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:45:59.934392 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928832    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:45:59.934392 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928835    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:45:59.934392 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928837    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:45:59.934392 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928840    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:45:59.934392 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928842    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:45:59.934392 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928845    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:45:59.934392 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928847    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:45:59.934392 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928850    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:45:59.934392 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928852    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:45:59.934392 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:45:59.928855    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:45:59.934811 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.928859    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:45:59.934811 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.928982    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 31 23:45:59.934811 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.931443    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 31 23:45:59.934811 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.933152    2578 server.go:1019] "Starting client certificate rotation"
Jul 31 23:45:59.934811 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.933254    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 23:45:59.934811 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.933281    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 23:45:59.936243 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.936229    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 23:45:59.940174 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.940151    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 23:45:59.951486 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.951470    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 31 23:45:59.953898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.953883    2578 log.go:25] "Validated CRI v1 image API"
Jul 31 23:45:59.954848 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.954832    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 31 23:45:59.957558 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.957539    2578 fs.go:135] Filesystem UUIDs: map[65046bcf-2459-4ec6-a2a8-8c06e69fc28c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 85da70c6-b23a-46b0-96e3-54f0f5950ac4:/dev/nvme0n1p4]
Jul 31 23:45:59.957606 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.957559    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 31 23:45:59.961765 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.961745    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 23:45:59.965412 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.965020    2578 manager.go:216] Machine: {Timestamp:2026-07-31 23:45:59.963522437 +0000 UTC m=+0.377086322 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104030 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ce30ce8910cdc402c49c845ed93fb SystemUUID:ec2ce30c-e891-0cdc-402c-49c845ed93fb BootID:b143242a-d79b-49f9-b427-937b97d1ec41 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:64:20:71:97:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:64:20:71:97:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:65:71:87:8c:5f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 31 23:45:59.965412 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.965409    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 31 23:45:59.965589 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.965496    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 31 23:45:59.970187 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.969980    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 31 23:45:59.970355 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.970194    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-3.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 31 23:45:59.971084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.971071    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 31 23:45:59.971146 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.971090    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 31 23:45:59.971146 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.971101    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 23:45:59.972027 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.972014    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 23:45:59.973280 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.973267    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 31 23:45:59.978603 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.978588    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 23:45:59.978811 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.978798    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 31 23:45:59.978877 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.978841    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 31 23:45:59.978877 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.978863    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 31 23:45:59.978965 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.978883    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 31 23:45:59.978965 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.978896    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 31 23:45:59.978965 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.978909    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 31 23:45:59.979340 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.979319    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 23:45:59.979400 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.979355    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 23:45:59.981739 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.981718    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 31 23:45:59.983499 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.983485    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 31 23:45:59.983542 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.983517    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 31 23:45:59.985529 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.985508    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2fqzw"
Jul 31 23:45:59.985656 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.985643    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 31 23:45:59.985691 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.985667    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 31 23:45:59.985691 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.985674    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 31 23:45:59.985691 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.985680    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 31 23:45:59.985691 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.985689    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 31 23:45:59.985800 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.985696    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 31 23:45:59.985800 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.985705    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 31 23:45:59.985800 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.985714    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 31 23:45:59.985800 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.985725    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 31 23:45:59.985800 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.985731    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 31 23:45:59.985800 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.985741    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 31 23:45:59.985800 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.985747    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 31 23:45:59.987423 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:45:59.987405    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-3.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 31 23:45:59.987479 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:45:59.987405    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 31 23:45:59.987553 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.987540    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 31 23:45:59.987586 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.987559    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 31 23:45:59.989753 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.989735    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-3.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 31 23:45:59.991026 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.991013    2578 server.go:1268] "Started kubelet"
Jul 31 23:45:59.991175 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.991122    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 31 23:45:59.991231 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.991200    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 31 23:45:59.991352 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.991324    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 31 23:45:59.991963 ip-10-0-140-3 systemd[1]: Started Kubernetes Kubelet.
Jul 31 23:45:59.992783 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.992713    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 31 23:45:59.993362 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.993341    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2fqzw"
Jul 31 23:45:59.993961 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:45:59.993949    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 31 23:45:59.997465 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:45:59.997441    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 31 23:45:59.998221 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:45:59.997095    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-3.ec2.internal.18c78496aef58b6f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-3.ec2.internal,UID:ip-10-0-140-3.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-3.ec2.internal,},FirstTimestamp:2026-07-31 23:45:59.990979439 +0000 UTC m=+0.404543308,LastTimestamp:2026-07-31 23:45:59.990979439 +0000 UTC m=+0.404543308,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-3.ec2.internal,}"
Jul 31 23:46:00.002027 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.002008    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 31 23:46:00.002141 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.002007    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 31 23:46:00.002199 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.002167    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found"
Jul 31 23:46:00.002888 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.002871    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 31 23:46:00.002974 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.002893    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 31 23:46:00.002974 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.002902    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 31 23:46:00.002974 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.002873    2578 factory.go:55] Registering systemd factory
Jul 31 23:46:00.002974 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.002965    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 31 23:46:00.003199 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.003001    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 31 23:46:00.003199 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.003007    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 31 23:46:00.003464 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.003450    2578 factory.go:153] Registering CRI-O factory
Jul 31 23:46:00.003464 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.003466    2578 factory.go:223] Registration of the crio container factory successfully
Jul 31 23:46:00.003596 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.003517    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 31 23:46:00.003596 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.003540    2578 factory.go:103] Registering Raw factory
Jul 31 23:46:00.003596 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.003556    2578 manager.go:1196] Started watching for new ooms in manager
Jul 31 23:46:00.004010 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.003969    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-3.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 31 23:46:00.004131 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.003995    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 31 23:46:00.004278 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.004254    2578 manager.go:349] Starting recovery of all containers
Jul 31 23:46:00.014591 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.014561    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 31 23:46:00.017008 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.016991    2578 manager.go:354] Recovery completed
Jul 31 23:46:00.021410 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.021395    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:00.023314 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.023297    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:00.023396 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.023328    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:00.023396 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.023360    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:00.023987 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.023974    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 31 23:46:00.023987 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.023986    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 31 23:46:00.024115 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.024001    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 23:46:00.024115 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.024019    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:00.024742 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.024727    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:00.024788 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.024754    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:00.024788 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.024763    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:00.026179 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.026166    2578 policy_none.go:49] "None policy: Start"
Jul 31 23:46:00.026241 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.026182    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 31 23:46:00.026241 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.026193    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 31 23:46:00.028255 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.028226    2578 policy_none.go:47] "Start"
Jul 31 23:46:00.065809 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.065655    2578 manager.go:337] "Starting Device Plugin manager"
Jul 31 23:46:00.065913 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.065838    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 31 23:46:00.065913 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.065852    2578 server.go:87] "Starting device plugin registration server"
Jul 31 23:46:00.066122 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.066110    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 31 23:46:00.066174 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.066123    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 31 23:46:00.066231 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.066215    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 31 23:46:00.066318 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.066300    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 31 23:46:00.066318 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.066314    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 31 23:46:00.067150 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.067129    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 31 23:46:00.067260 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.067183    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-3.ec2.internal\" not found"
Jul 31 23:46:00.125036 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.125004    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 31 23:46:00.127212 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.125058    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 31 23:46:00.127212 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.125082    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 31 23:46:00.127212 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.125119    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 31 23:46:00.127385 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.127371    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:00.166679 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.166594    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:00.167568 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.167547    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:00.167660 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.167591    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:00.167660 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.167600    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:00.167660 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.167626    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-3.ec2.internal"
Jul 31 23:46:00.176411 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.176396    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-3.ec2.internal"
Jul 31 23:46:00.176469 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.176419    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-3.ec2.internal\": node \"ip-10-0-140-3.ec2.internal\" not found"
Jul 31 23:46:00.190359 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.190331    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found"
Jul 31 23:46:00.225673 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.225645    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal"]
Jul 31 23:46:00.225765 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.225710    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:00.227308 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.227288    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:00.227414 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.227322    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:00.227414 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.227332    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:00.229637 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.229625    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:00.229739 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.229724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal"
Jul 31 23:46:00.229802 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.229759    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:00.230436 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.230419    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:00.230436 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.230431    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:00.230565 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.230452    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:00.230565 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.230464    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:00.230565 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.230452    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:00.230565 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.230543    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:00.232797 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.232782    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal"
Jul 31 23:46:00.232852 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.232806    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:00.233485 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.233462    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:00.233485 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.233486    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:00.233586 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.233497    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:00.258714 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.258689    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-3.ec2.internal\" not found" node="ip-10-0-140-3.ec2.internal"
Jul 31 23:46:00.263209 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.263195    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-3.ec2.internal\" not found" node="ip-10-0-140-3.ec2.internal"
Jul 31 23:46:00.291162 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.291144    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found"
Jul 31 23:46:00.304876 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.304852    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d96382247de37d9245ec69df207c39f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal\" (UID: \"d96382247de37d9245ec69df207c39f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal"
Jul 31 23:46:00.304970 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.304884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8a89e13fce40802bf99f22858790a870-config\") pod \"kube-apiserver-proxy-ip-10-0-140-3.ec2.internal\" (UID: \"8a89e13fce40802bf99f22858790a870\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal"
Jul 31 23:46:00.304970 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.304902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d96382247de37d9245ec69df207c39f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal\" (UID: \"d96382247de37d9245ec69df207c39f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal"
Jul 31 23:46:00.392213 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.392167    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found"
Jul 31 23:46:00.405585 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.405561    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d96382247de37d9245ec69df207c39f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal\" (UID: \"d96382247de37d9245ec69df207c39f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal"
Jul 31 23:46:00.405666 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.405526    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d96382247de37d9245ec69df207c39f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal\" (UID: \"d96382247de37d9245ec69df207c39f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal"
Jul 31 23:46:00.405666 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.405654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8a89e13fce40802bf99f22858790a870-config\") pod \"kube-apiserver-proxy-ip-10-0-140-3.ec2.internal\" (UID: \"8a89e13fce40802bf99f22858790a870\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal"
Jul 31 23:46:00.405732 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.405672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d96382247de37d9245ec69df207c39f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal\" (UID: \"d96382247de37d9245ec69df207c39f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal"
Jul 31 23:46:00.405732 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.405696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d96382247de37d9245ec69df207c39f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal\" (UID: \"d96382247de37d9245ec69df207c39f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal"
Jul 31 23:46:00.405732 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.405721    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8a89e13fce40802bf99f22858790a870-config\") pod \"kube-apiserver-proxy-ip-10-0-140-3.ec2.internal\" (UID: \"8a89e13fce40802bf99f22858790a870\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal"
Jul 31 23:46:00.493012 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.492945    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found"
Jul 31 23:46:00.561471 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.561449    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal"
Jul 31 23:46:00.565731 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.565707    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal"
Jul 31 23:46:00.593419 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.593390    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found"
Jul 31 23:46:00.694009 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.693961    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found"
Jul 31 23:46:00.794560 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.794474    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found"
Jul 31 23:46:00.866830 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.866796    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:00.895139 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.895102    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found"
Jul 31 23:46:00.932588 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.932555    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 31 23:46:00.933098 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.932749    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 23:46:00.933098 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:00.932753    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 23:46:00.995545 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:00.995518    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found"
Jul 31 23:46:01.000500 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:01.000469    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 23:41:01 +0000 UTC" deadline="2028-02-17 19:08:25.915188142 +0000 UTC"
Jul 31 23:46:01.000500 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:01.000496    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13579h22m24.914695105s"
Jul 31 23:46:01.096626 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:01.096547    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found"
Jul 31 23:46:01.166907 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:46:01.166794    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a89e13fce40802bf99f22858790a870.slice/crio-f8d0e71a8def1c0aae33c6addbbb27aec6e45b2c0623a6fe2095380796db18e5 WatchSource:0}: Error finding container f8d0e71a8def1c0aae33c6addbbb27aec6e45b2c0623a6fe2095380796db18e5: Status 404 returned error can't find the container with id f8d0e71a8def1c0aae33c6addbbb27aec6e45b2c0623a6fe2095380796db18e5
Jul 31 23:46:01.167603 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:46:01.167570    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96382247de37d9245ec69df207c39f4.slice/crio-5f10c21bcee852d9bfed42bcd8be3afd8355cb79417837ade5fb6b18d3f96589 WatchSource:0}: Error finding container 5f10c21bcee852d9bfed42bcd8be3afd8355cb79417837ade5fb6b18d3f96589: Status 404 returned error can't find the container with id 5f10c21bcee852d9bfed42bcd8be3afd8355cb79417837ade5fb6b18d3f96589
Jul 31 23:46:01.174304 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:01.174288    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 23:46:01.196871 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:01.196838    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found"
Jul 31 23:46:01.224933 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:01.224911    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:01.302369 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:01.302341    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal"
Jul 31 23:46:01.311750 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:01.311735    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 23:46:01.312749 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:01.312737    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal"
Jul 31 23:46:01.321726 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:01.321712    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 23:46:02.517834 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:02.517813    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:03.034265 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.034238    2578 apiserver.go:52] "Watching apiserver"
Jul 31 23:46:03.039309 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.039285    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 31 23:46:03.039756 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.039728    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-zzsl6","kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6","openshift-cluster-node-tuning-operator/tuned-pv7nj","openshift-image-registry/node-ca-jxfkn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal","openshift-multus/network-metrics-daemon-js2f5","openshift-network-operator/iptables-alerter-pzp69","kube-system/konnectivity-agent-7ng6n","openshift-dns/node-resolver-5r99m","openshift-multus/multus-additional-cni-plugins-6jv4r","openshift-multus/multus-ppq4j","openshift-network-diagnostics/network-check-target-mjkm4","openshift-ovn-kubernetes/ovnkube-node-xdzrq"]
Jul 31 23:46:03.042479 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.042454    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:03.042571 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.042535    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:03.044902 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.044881    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:03.044998 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.044955    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:03.047143 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.047126    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.049177 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.049156    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.050174 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.050157    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-299z5\""
Jul 31 23:46:03.050435 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.050420    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 31 23:46:03.050660 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.050644    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 31 23:46:03.050827 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.050804    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 31 23:46:03.051046 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.051033    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:03.051216 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.051201    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:03.052110 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.051341    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5cdj6\""
Jul 31 23:46:03.052110 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.051545    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jxfkn"
Jul 31 23:46:03.053100 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.053082    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 31 23:46:03.053238 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.053220    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 31 23:46:03.053394 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.053378    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-84w28\""
Jul 31 23:46:03.053679 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.053662    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.054217 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.053799    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 31 23:46:03.055400 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.055380    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 31 23:46:03.055890 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.055736    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 31 23:46:03.056070 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.056050    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7ng6n"
Jul 31 23:46:03.056174 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.056158    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 31 23:46:03.057194 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.057173    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 31 23:46:03.057312 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.057294    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2pkbp\""
Jul 31 23:46:03.057460 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.057447    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 31 23:46:03.057532 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.057517    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 31 23:46:03.057578 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.057556    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 31 23:46:03.058236 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.058219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:03.058340 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.058319    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:03.059545 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.059377    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 31 23:46:03.059545 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.059396    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 31 23:46:03.060736 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.060698    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pzp69"
Jul 31 23:46:03.061200 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.061183    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x97fg\""
Jul 31 23:46:03.062532 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.062501    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:03.062902 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.062887    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:03.064959 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.063289    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 31 23:46:03.064959 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.063883    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hwbp8\""
Jul 31 23:46:03.064959 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.064314    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5r99m"
Jul 31 23:46:03.067131 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.067105    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 31 23:46:03.068225 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.068206    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.069060 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.068992    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sktv6\""
Jul 31 23:46:03.069893 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.069872    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 31 23:46:03.070900 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.070877    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 23:46:03.071926 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.070955    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 31 23:46:03.071926 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.071376    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.071926 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.071454    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vfmzb\""
Jul 31 23:46:03.071926 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.071636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pqmh\" (UniqueName: \"kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh\") pod \"network-check-target-mjkm4\" (UID: \"824b4cc2-7b16-43c6-9cc0-46759ffdf9b5\") " pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:03.071926 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.071706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-systemd-units\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.071926 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.071759    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.071926 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.071816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9c7x\" (UniqueName: \"kubernetes.io/projected/a87b1f31-398c-45b4-a99c-5115633e0fc3-kube-api-access-g9c7x\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.071926 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.071856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1ddbfa22-7b8f-4800-8dd1-78a871bbf08d-agent-certs\") pod \"konnectivity-agent-7ng6n\" (UID: \"1ddbfa22-7b8f-4800-8dd1-78a871bbf08d\") " pod="kube-system/konnectivity-agent-7ng6n"
Jul 31 23:46:03.071926 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.071887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1ddbfa22-7b8f-4800-8dd1-78a871bbf08d-konnectivity-ca\") pod \"konnectivity-agent-7ng6n\" (UID: \"1ddbfa22-7b8f-4800-8dd1-78a871bbf08d\") " pod="kube-system/konnectivity-agent-7ng6n"
Jul 31 23:46:03.071926 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.071917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fdd94922-bfea-4c24-ac38-66daa163d31c-sys-fs\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.072498 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.071945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-modprobe-d\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.072498 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.072126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-kubernetes\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.072498 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.072198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-run\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.072498 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.072263    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 31 23:46:03.072498 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.072284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-lib-modules\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.072498 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.072415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-run-systemd\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.072962 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.072899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-run-openvswitch\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.073107 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.073084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-ovnkube-config\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.073194 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.073129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a87b1f31-398c-45b4-a99c-5115633e0fc3-tmp\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.073779 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.073753    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 31 23:46:03.073843 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.073821    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 31 23:46:03.075135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.073959    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 31 23:46:03.075135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.073198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs\") pod \"network-metrics-daemon-js2f5\" (UID: \"ec95cdf0-c411-409a-a0e5-274712ef6c00\") " pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:03.075135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fdd94922-bfea-4c24-ac38-66daa163d31c-device-dir\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.075135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fdd94922-bfea-4c24-ac38-66daa163d31c-etc-selinux\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.075135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-sysconfig\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.075135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-var-lib-kubelet\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.075135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-node-log\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.075135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074293    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9q92m\""
Jul 31 23:46:03.075135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074308    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 31 23:46:03.075135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-cni-bin\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.075135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d49s9\" (UniqueName: \"kubernetes.io/projected/ec95cdf0-c411-409a-a0e5-274712ef6c00-kube-api-access-d49s9\") pod \"network-metrics-daemon-js2f5\" (UID: \"ec95cdf0-c411-409a-a0e5-274712ef6c00\") " pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:03.075135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-sysctl-d\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.075135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-sysctl-conf\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.075135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074418    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c459a9c2-d75c-44d9-865b-4262edeb4f61-host\") pod \"node-ca-jxfkn\" (UID: \"c459a9c2-d75c-44d9-865b-4262edeb4f61\") " pod="openshift-image-registry/node-ca-jxfkn"
Jul 31 23:46:03.075135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074439    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c459a9c2-d75c-44d9-865b-4262edeb4f61-serviceca\") pod \"node-ca-jxfkn\" (UID: \"c459a9c2-d75c-44d9-865b-4262edeb4f61\") " pod="openshift-image-registry/node-ca-jxfkn"
Jul 31 23:46:03.075135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsrlg\" (UniqueName: \"kubernetes.io/projected/c459a9c2-d75c-44d9-865b-4262edeb4f61-kube-api-access-bsrlg\") pod \"node-ca-jxfkn\" (UID: \"c459a9c2-d75c-44d9-865b-4262edeb4f61\") " pod="openshift-image-registry/node-ca-jxfkn"
Jul 31 23:46:03.075135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab5415d7-3b45-4861-bfaf-be67c4d16871-kubelet-config\") pod \"global-pull-secret-syncer-zzsl6\" (UID: \"ab5415d7-3b45-4861-bfaf-be67c4d16871\") " pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:03.075135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab5415d7-3b45-4861-bfaf-be67c4d16871-dbus\") pod \"global-pull-secret-syncer-zzsl6\" (UID: \"ab5415d7-3b45-4861-bfaf-be67c4d16871\") " pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:03.075839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fdd94922-bfea-4c24-ac38-66daa163d31c-registration-dir\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.075839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074556    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-systemd\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.075839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-var-lib-openvswitch\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.075839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-env-overrides\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.075839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-ovn-node-metrics-cert\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.075839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-ovnkube-script-lib\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.075839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074665    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nz2n\" (UniqueName: \"kubernetes.io/projected/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-kube-api-access-2nz2n\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.075839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret\") pod \"global-pull-secret-syncer-zzsl6\" (UID: \"ab5415d7-3b45-4861-bfaf-be67c4d16871\") " pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:03.075839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fdd94922-bfea-4c24-ac38-66daa163d31c-socket-dir\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.075839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-sys\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.075839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074775    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-slash\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.075839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-run-netns\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.075839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-etc-openvswitch\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.075839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-cni-netd\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.075839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fdd94922-bfea-4c24-ac38-66daa163d31c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.075839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6xnr\" (UniqueName: \"kubernetes.io/projected/fdd94922-bfea-4c24-ac38-66daa163d31c-kube-api-access-x6xnr\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.076553 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-host\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.076553 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-tuned\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.076553 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074928    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-kubelet\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.076553 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-run-ovn\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.076553 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.074982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-log-socket\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.076553 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.075008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.096850 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.096824    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v8flw"
Jul 31 23:46:03.105801 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.105779    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v8flw"
Jul 31 23:46:03.157665 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.157631    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 31 23:46:03.175252 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1ddbfa22-7b8f-4800-8dd1-78a871bbf08d-agent-certs\") pod \"konnectivity-agent-7ng6n\" (UID: \"1ddbfa22-7b8f-4800-8dd1-78a871bbf08d\") " pod="kube-system/konnectivity-agent-7ng6n"
Jul 31 23:46:03.175428 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1ddbfa22-7b8f-4800-8dd1-78a871bbf08d-konnectivity-ca\") pod \"konnectivity-agent-7ng6n\" (UID: \"1ddbfa22-7b8f-4800-8dd1-78a871bbf08d\") " pod="kube-system/konnectivity-agent-7ng6n"
Jul 31 23:46:03.175428 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-cnibin\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.175428 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-os-release\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.175428 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-host-run-k8s-cni-cncf-io\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.175428 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-host-var-lib-cni-bin\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.175428 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-multus-conf-dir\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.175728 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fdd94922-bfea-4c24-ac38-66daa163d31c-sys-fs\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.175728 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175474    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-kubernetes\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.175728 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-lib-modules\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.175728 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175559    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fdd94922-bfea-4c24-ac38-66daa163d31c-sys-fs\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.175728 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-kubernetes\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.175728 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nz2n\" (UniqueName: \"kubernetes.io/projected/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-kube-api-access-2nz2n\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.175728 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175648    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-multus-socket-dir-parent\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.176037 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175746    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-lib-modules\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.176037 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs\") pod \"network-metrics-daemon-js2f5\" (UID: \"ec95cdf0-c411-409a-a0e5-274712ef6c00\") " pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:03.176037 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fdd94922-bfea-4c24-ac38-66daa163d31c-device-dir\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.176037 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-cni-bin\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.176037 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175839    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 31 23:46:03.176037 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7405cb77-3fa3-47ae-864d-b8b59f586932-cnibin\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.176037 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175911    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7405cb77-3fa3-47ae-864d-b8b59f586932-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.176037 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6s2r\" (UniqueName: \"kubernetes.io/projected/7405cb77-3fa3-47ae-864d-b8b59f586932-kube-api-access-w6s2r\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.176037 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-sysctl-d\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.176037 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.175997    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:03.176448 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176057    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1ddbfa22-7b8f-4800-8dd1-78a871bbf08d-konnectivity-ca\") pod \"konnectivity-agent-7ng6n\" (UID: \"1ddbfa22-7b8f-4800-8dd1-78a871bbf08d\") " pod="kube-system/konnectivity-agent-7ng6n"
Jul 31 23:46:03.176448 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.176075    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs podName:ec95cdf0-c411-409a-a0e5-274712ef6c00 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:03.676039592 +0000 UTC m=+3.035930614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs") pod "network-metrics-daemon-js2f5" (UID: "ec95cdf0-c411-409a-a0e5-274712ef6c00") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:03.176448 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab5415d7-3b45-4861-bfaf-be67c4d16871-dbus\") pod \"global-pull-secret-syncer-zzsl6\" (UID: \"ab5415d7-3b45-4861-bfaf-be67c4d16871\") " pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:03.176448 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.175997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab5415d7-3b45-4861-bfaf-be67c4d16871-dbus\") pod \"global-pull-secret-syncer-zzsl6\" (UID: \"ab5415d7-3b45-4861-bfaf-be67c4d16871\") " pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:03.176448 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-host-run-netns\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.176448 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-systemd\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.176448 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176257    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-sysctl-d\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.176448 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-var-lib-openvswitch\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.176448 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-ovn-node-metrics-cert\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.176448 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-cni-bin\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.176448 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fdd94922-bfea-4c24-ac38-66daa163d31c-device-dir\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.176448 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-ovnkube-script-lib\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.176448 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-systemd\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.177151 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176489    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-var-lib-openvswitch\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.177151 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7405cb77-3fa3-47ae-864d-b8b59f586932-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.177151 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42d1f0a2-5cd8-4cef-a805-f489af2e7931-host-slash\") pod \"iptables-alerter-pzp69\" (UID: \"42d1f0a2-5cd8-4cef-a805-f489af2e7931\") " pod="openshift-network-operator/iptables-alerter-pzp69"
Jul 31 23:46:03.177151 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176681    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-host-var-lib-kubelet\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.177151 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pztkg\" (UniqueName: \"kubernetes.io/projected/d1a3dcea-7f41-400b-b402-9cf780a6b030-kube-api-access-pztkg\") pod \"node-resolver-5r99m\" (UID: \"d1a3dcea-7f41-400b-b402-9cf780a6b030\") " pod="openshift-dns/node-resolver-5r99m"
Jul 31 23:46:03.177151 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-sys\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.177151 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-etc-openvswitch\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.177151 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176797    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6xnr\" (UniqueName: \"kubernetes.io/projected/fdd94922-bfea-4c24-ac38-66daa163d31c-kube-api-access-x6xnr\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.177151 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-tuned\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.177151 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176861    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-log-socket\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.177151 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.177151 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.176955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pqmh\" (UniqueName: \"kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh\") pod \"network-check-target-mjkm4\" (UID: \"824b4cc2-7b16-43c6-9cc0-46759ffdf9b5\") " pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:03.177151 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.177010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.177151 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.177041    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-log-socket\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.177818 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.177166    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-ovnkube-script-lib\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.177818 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.177629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7405cb77-3fa3-47ae-864d-b8b59f586932-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.177818 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.177707    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.177818 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.177795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-sys\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.178003 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.177860    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-etc-openvswitch\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.178003 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.177909    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a3dcea-7f41-400b-b402-9cf780a6b030-tmp-dir\") pod \"node-resolver-5r99m\" (UID: \"d1a3dcea-7f41-400b-b402-9cf780a6b030\") " pod="openshift-dns/node-resolver-5r99m"
Jul 31 23:46:03.178003 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.177929    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.178003 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.177978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9c7x\" (UniqueName: \"kubernetes.io/projected/a87b1f31-398c-45b4-a99c-5115633e0fc3-kube-api-access-g9c7x\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.178188 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-modprobe-d\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.178188 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178041    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-run\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.178188 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-run-systemd\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.178188 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-run-openvswitch\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.178188 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-ovnkube-config\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.178188 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-etc-kubernetes\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.178188 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a87b1f31-398c-45b4-a99c-5115633e0fc3-tmp\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.178476 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fdd94922-bfea-4c24-ac38-66daa163d31c-etc-selinux\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.178476 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-sysconfig\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.178476 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-var-lib-kubelet\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.178476 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-modprobe-d\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.178476 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-node-log\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.178476 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab5415d7-3b45-4861-bfaf-be67c4d16871-kubelet-config\") pod \"global-pull-secret-syncer-zzsl6\" (UID: \"ab5415d7-3b45-4861-bfaf-be67c4d16871\") " pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:03.178476 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-run-openvswitch\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.178476 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178382    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-run-systemd\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.178476 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-var-lib-kubelet\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.178476 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fdd94922-bfea-4c24-ac38-66daa163d31c-etc-selinux\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.178960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-multus-cni-dir\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.178960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2psw\" (UniqueName: \"kubernetes.io/projected/345890db-ca81-4c74-a721-ba0904c50bf4-kube-api-access-p2psw\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.178960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d49s9\" (UniqueName: \"kubernetes.io/projected/ec95cdf0-c411-409a-a0e5-274712ef6c00-kube-api-access-d49s9\") pod \"network-metrics-daemon-js2f5\" (UID: \"ec95cdf0-c411-409a-a0e5-274712ef6c00\") " pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:03.178960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-sysctl-conf\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.178960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-sysconfig\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.178960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c459a9c2-d75c-44d9-865b-4262edeb4f61-host\") pod \"node-ca-jxfkn\" (UID: \"c459a9c2-d75c-44d9-865b-4262edeb4f61\") " pod="openshift-image-registry/node-ca-jxfkn"
Jul 31 23:46:03.178960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c459a9c2-d75c-44d9-865b-4262edeb4f61-serviceca\") pod \"node-ca-jxfkn\" (UID: \"c459a9c2-d75c-44d9-865b-4262edeb4f61\") " pod="openshift-image-registry/node-ca-jxfkn"
Jul 31 23:46:03.178960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsrlg\" (UniqueName: \"kubernetes.io/projected/c459a9c2-d75c-44d9-865b-4262edeb4f61-kube-api-access-bsrlg\") pod \"node-ca-jxfkn\" (UID: \"c459a9c2-d75c-44d9-865b-4262edeb4f61\") " pod="openshift-image-registry/node-ca-jxfkn"
Jul 31 23:46:03.178960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret\") pod \"global-pull-secret-syncer-zzsl6\" (UID: \"ab5415d7-3b45-4861-bfaf-be67c4d16871\") " pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:03.178960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178766    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7405cb77-3fa3-47ae-864d-b8b59f586932-system-cni-dir\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.178960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-system-cni-dir\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.178960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fdd94922-bfea-4c24-ac38-66daa163d31c-registration-dir\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.178960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-env-overrides\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.178960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/42d1f0a2-5cd8-4cef-a805-f489af2e7931-iptables-alerter-script\") pod \"iptables-alerter-pzp69\" (UID: \"42d1f0a2-5cd8-4cef-a805-f489af2e7931\") " pod="openshift-network-operator/iptables-alerter-pzp69"
Jul 31 23:46:03.178960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-host-var-lib-cni-multus\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.178960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-hostroot\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.179669 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178984    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-ovnkube-config\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.179669 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.179029    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-node-log\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.179669 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.179233    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c459a9c2-d75c-44d9-865b-4262edeb4f61-host\") pod \"node-ca-jxfkn\" (UID: \"c459a9c2-d75c-44d9-865b-4262edeb4f61\") " pod="openshift-image-registry/node-ca-jxfkn"
Jul 31 23:46:03.179669 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.179638    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c459a9c2-d75c-44d9-865b-4262edeb4f61-serviceca\") pod \"node-ca-jxfkn\" (UID: \"c459a9c2-d75c-44d9-865b-4262edeb4f61\") " pod="openshift-image-registry/node-ca-jxfkn"
Jul 31 23:46:03.179894 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.179694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fdd94922-bfea-4c24-ac38-66daa163d31c-registration-dir\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.179894 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.179787    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-ovn-node-metrics-cert\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.179894 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.178991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d1a3dcea-7f41-400b-b402-9cf780a6b030-hosts-file\") pod \"node-resolver-5r99m\" (UID: \"d1a3dcea-7f41-400b-b402-9cf780a6b030\") " pod="openshift-dns/node-resolver-5r99m"
Jul 31 23:46:03.179894 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.179795    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:03.179894 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.179876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fdd94922-bfea-4c24-ac38-66daa163d31c-socket-dir\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.180161 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.179905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-slash\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.180161 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.179932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-run-netns\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.180161 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.179906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-sysctl-conf\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.180161 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.179948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1ddbfa22-7b8f-4800-8dd1-78a871bbf08d-agent-certs\") pod \"konnectivity-agent-7ng6n\" (UID: \"1ddbfa22-7b8f-4800-8dd1-78a871bbf08d\") " pod="kube-system/konnectivity-agent-7ng6n"
Jul 31 23:46:03.180161 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.179963    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-cni-netd\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.180161 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.179994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xhm4\" (UniqueName: \"kubernetes.io/projected/42d1f0a2-5cd8-4cef-a805-f489af2e7931-kube-api-access-4xhm4\") pod \"iptables-alerter-pzp69\" (UID: \"42d1f0a2-5cd8-4cef-a805-f489af2e7931\") " pod="openshift-network-operator/iptables-alerter-pzp69"
Jul 31 23:46:03.180161 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.180018    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-slash\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.180161 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.180033    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret podName:ab5415d7-3b45-4861-bfaf-be67c4d16871 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:03.680004159 +0000 UTC m=+3.039895184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret") pod "global-pull-secret-syncer-zzsl6" (UID: "ab5415d7-3b45-4861-bfaf-be67c4d16871") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:03.180161 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.180087    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab5415d7-3b45-4861-bfaf-be67c4d16871-kubelet-config\") pod \"global-pull-secret-syncer-zzsl6\" (UID: \"ab5415d7-3b45-4861-bfaf-be67c4d16871\") " pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:03.180161 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.180092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-cni-netd\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.180161 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.180089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/345890db-ca81-4c74-a721-ba0904c50bf4-cni-binary-copy\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.180161 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.180122    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-env-overrides\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.180161 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.180130    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-run-netns\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.180618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.180181    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-run\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.180618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.180210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/345890db-ca81-4c74-a721-ba0904c50bf4-multus-daemon-config\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.180618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.180243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fdd94922-bfea-4c24-ac38-66daa163d31c-socket-dir\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.180618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.180253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fdd94922-bfea-4c24-ac38-66daa163d31c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.180618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.180290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-host\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.180618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.180326    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-kubelet\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.180618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.180347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fdd94922-bfea-4c24-ac38-66daa163d31c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.180618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.180358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-run-ovn\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.180618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.180394    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a87b1f31-398c-45b4-a99c-5115633e0fc3-host\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.180618 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.180420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7405cb77-3fa3-47ae-864d-b8b59f586932-cni-binary-copy\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.181610 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.181582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-host-run-multus-certs\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.181744 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.181637    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-systemd-units\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.181744 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.181670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7405cb77-3fa3-47ae-864d-b8b59f586932-os-release\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.181860 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.181822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a87b1f31-398c-45b4-a99c-5115633e0fc3-etc-tuned\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.181924 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.181910    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-systemd-units\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.181977 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.181949    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-host-kubelet\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.182028 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.181985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-run-ovn\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.182458 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.182440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a87b1f31-398c-45b4-a99c-5115633e0fc3-tmp\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.184020 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.184000    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:03.184116 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.184023    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:03.184116 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.184036    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9pqmh for pod openshift-network-diagnostics/network-check-target-mjkm4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:03.184224 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.184118    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh podName:824b4cc2-7b16-43c6-9cc0-46759ffdf9b5 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:03.684090499 +0000 UTC m=+3.043981523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9pqmh" (UniqueName: "kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh") pod "network-check-target-mjkm4" (UID: "824b4cc2-7b16-43c6-9cc0-46759ffdf9b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:03.185524 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.185494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6xnr\" (UniqueName: \"kubernetes.io/projected/fdd94922-bfea-4c24-ac38-66daa163d31c-kube-api-access-x6xnr\") pod \"aws-ebs-csi-driver-node-kgtw6\" (UID: \"fdd94922-bfea-4c24-ac38-66daa163d31c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.186096 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.186074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nz2n\" (UniqueName: \"kubernetes.io/projected/030ae37c-094e-4a10-bea2-8e2b8bcc6d5e-kube-api-access-2nz2n\") pod \"ovnkube-node-xdzrq\" (UID: \"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.188130 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.188111    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9c7x\" (UniqueName: \"kubernetes.io/projected/a87b1f31-398c-45b4-a99c-5115633e0fc3-kube-api-access-g9c7x\") pod \"tuned-pv7nj\" (UID: \"a87b1f31-398c-45b4-a99c-5115633e0fc3\") " pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.188640 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.188595    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal" event={"ID":"8a89e13fce40802bf99f22858790a870","Type":"ContainerStarted","Data":"f8d0e71a8def1c0aae33c6addbbb27aec6e45b2c0623a6fe2095380796db18e5"}
Jul 31 23:46:03.188893 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.188873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsrlg\" (UniqueName: \"kubernetes.io/projected/c459a9c2-d75c-44d9-865b-4262edeb4f61-kube-api-access-bsrlg\") pod \"node-ca-jxfkn\" (UID: \"c459a9c2-d75c-44d9-865b-4262edeb4f61\") " pod="openshift-image-registry/node-ca-jxfkn"
Jul 31 23:46:03.189810 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.189785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal" event={"ID":"d96382247de37d9245ec69df207c39f4","Type":"ContainerStarted","Data":"5f10c21bcee852d9bfed42bcd8be3afd8355cb79417837ade5fb6b18d3f96589"}
Jul 31 23:46:03.194622 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.194589    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d49s9\" (UniqueName: \"kubernetes.io/projected/ec95cdf0-c411-409a-a0e5-274712ef6c00-kube-api-access-d49s9\") pod \"network-metrics-daemon-js2f5\" (UID: \"ec95cdf0-c411-409a-a0e5-274712ef6c00\") " pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:03.234693 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.234659    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:03.283101 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-cnibin\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.283101 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-os-release\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.283344 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-host-run-k8s-cni-cncf-io\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.283344 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-host-var-lib-cni-bin\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.283344 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-multus-conf-dir\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.283344 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-cnibin\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.283344 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283182    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-multus-socket-dir-parent\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.283344 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283223    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-multus-socket-dir-parent\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.283344 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7405cb77-3fa3-47ae-864d-b8b59f586932-cnibin\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.283344 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-host-run-k8s-cni-cncf-io\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.283344 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283261    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-host-var-lib-cni-bin\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.283344 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7405cb77-3fa3-47ae-864d-b8b59f586932-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.283344 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283292    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-multus-conf-dir\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.283344 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283299    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7405cb77-3fa3-47ae-864d-b8b59f586932-cnibin\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.283344 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283221    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-os-release\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.283344 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6s2r\" (UniqueName: \"kubernetes.io/projected/7405cb77-3fa3-47ae-864d-b8b59f586932-kube-api-access-w6s2r\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.283344 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-host-run-netns\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.284084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283379    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7405cb77-3fa3-47ae-864d-b8b59f586932-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.284084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42d1f0a2-5cd8-4cef-a805-f489af2e7931-host-slash\") pod \"iptables-alerter-pzp69\" (UID: \"42d1f0a2-5cd8-4cef-a805-f489af2e7931\") " pod="openshift-network-operator/iptables-alerter-pzp69"
Jul 31 23:46:03.284084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-host-var-lib-kubelet\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.284084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283436    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pztkg\" (UniqueName: \"kubernetes.io/projected/d1a3dcea-7f41-400b-b402-9cf780a6b030-kube-api-access-pztkg\") pod \"node-resolver-5r99m\" (UID: \"d1a3dcea-7f41-400b-b402-9cf780a6b030\") " pod="openshift-dns/node-resolver-5r99m"
Jul 31 23:46:03.284084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283456    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7405cb77-3fa3-47ae-864d-b8b59f586932-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.284084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7405cb77-3fa3-47ae-864d-b8b59f586932-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.284084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283490    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42d1f0a2-5cd8-4cef-a805-f489af2e7931-host-slash\") pod \"iptables-alerter-pzp69\" (UID: \"42d1f0a2-5cd8-4cef-a805-f489af2e7931\") " pod="openshift-network-operator/iptables-alerter-pzp69"
Jul 31 23:46:03.284084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a3dcea-7f41-400b-b402-9cf780a6b030-tmp-dir\") pod \"node-resolver-5r99m\" (UID: \"d1a3dcea-7f41-400b-b402-9cf780a6b030\") " pod="openshift-dns/node-resolver-5r99m"
Jul 31 23:46:03.284084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-host-run-netns\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.284084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-etc-kubernetes\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.284084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283542    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-etc-kubernetes\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.284084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-multus-cni-dir\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.284084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283581    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2psw\" (UniqueName: \"kubernetes.io/projected/345890db-ca81-4c74-a721-ba0904c50bf4-kube-api-access-p2psw\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.284084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7405cb77-3fa3-47ae-864d-b8b59f586932-system-cni-dir\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.284084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-system-cni-dir\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.284084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283677    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/42d1f0a2-5cd8-4cef-a805-f489af2e7931-iptables-alerter-script\") pod \"iptables-alerter-pzp69\" (UID: \"42d1f0a2-5cd8-4cef-a805-f489af2e7931\") " pod="openshift-network-operator/iptables-alerter-pzp69"
Jul 31 23:46:03.284084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-host-var-lib-cni-multus\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.284954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283702    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-host-var-lib-kubelet\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.284954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-hostroot\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.284954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d1a3dcea-7f41-400b-b402-9cf780a6b030-hosts-file\") pod \"node-resolver-5r99m\" (UID: \"d1a3dcea-7f41-400b-b402-9cf780a6b030\") " pod="openshift-dns/node-resolver-5r99m"
Jul 31 23:46:03.284954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xhm4\" (UniqueName: \"kubernetes.io/projected/42d1f0a2-5cd8-4cef-a805-f489af2e7931-kube-api-access-4xhm4\") pod \"iptables-alerter-pzp69\" (UID: \"42d1f0a2-5cd8-4cef-a805-f489af2e7931\") " pod="openshift-network-operator/iptables-alerter-pzp69"
Jul 31 23:46:03.284954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/345890db-ca81-4c74-a721-ba0904c50bf4-cni-binary-copy\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.284954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/345890db-ca81-4c74-a721-ba0904c50bf4-multus-daemon-config\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.284954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7405cb77-3fa3-47ae-864d-b8b59f586932-cni-binary-copy\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.284954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-host-run-multus-certs\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.284954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7405cb77-3fa3-47ae-864d-b8b59f586932-os-release\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.284954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.283983    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7405cb77-3fa3-47ae-864d-b8b59f586932-os-release\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.284954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.284004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7405cb77-3fa3-47ae-864d-b8b59f586932-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.284954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.284024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7405cb77-3fa3-47ae-864d-b8b59f586932-system-cni-dir\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.284954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.284058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-system-cni-dir\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.284954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.284076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-multus-cni-dir\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.284954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.284189    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a3dcea-7f41-400b-b402-9cf780a6b030-tmp-dir\") pod \"node-resolver-5r99m\" (UID: \"d1a3dcea-7f41-400b-b402-9cf780a6b030\") " pod="openshift-dns/node-resolver-5r99m"
Jul 31 23:46:03.284954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.284267    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-host-var-lib-cni-multus\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.284954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.284315    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-hostroot\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.284954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.284476    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/42d1f0a2-5cd8-4cef-a805-f489af2e7931-iptables-alerter-script\") pod \"iptables-alerter-pzp69\" (UID: \"42d1f0a2-5cd8-4cef-a805-f489af2e7931\") " pod="openshift-network-operator/iptables-alerter-pzp69"
Jul 31 23:46:03.285823 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.284548    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d1a3dcea-7f41-400b-b402-9cf780a6b030-hosts-file\") pod \"node-resolver-5r99m\" (UID: \"d1a3dcea-7f41-400b-b402-9cf780a6b030\") " pod="openshift-dns/node-resolver-5r99m"
Jul 31 23:46:03.285823 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.284598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/345890db-ca81-4c74-a721-ba0904c50bf4-host-run-multus-certs\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.285823 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.284670    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/345890db-ca81-4c74-a721-ba0904c50bf4-multus-daemon-config\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.285823 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.284879    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7405cb77-3fa3-47ae-864d-b8b59f586932-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.285823 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.284952    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7405cb77-3fa3-47ae-864d-b8b59f586932-cni-binary-copy\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.285823 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.284986    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/345890db-ca81-4c74-a721-ba0904c50bf4-cni-binary-copy\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.293837 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.293809    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pztkg\" (UniqueName: \"kubernetes.io/projected/d1a3dcea-7f41-400b-b402-9cf780a6b030-kube-api-access-pztkg\") pod \"node-resolver-5r99m\" (UID: \"d1a3dcea-7f41-400b-b402-9cf780a6b030\") " pod="openshift-dns/node-resolver-5r99m"
Jul 31 23:46:03.294111 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.294080    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xhm4\" (UniqueName: \"kubernetes.io/projected/42d1f0a2-5cd8-4cef-a805-f489af2e7931-kube-api-access-4xhm4\") pod \"iptables-alerter-pzp69\" (UID: \"42d1f0a2-5cd8-4cef-a805-f489af2e7931\") " pod="openshift-network-operator/iptables-alerter-pzp69"
Jul 31 23:46:03.294327 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.294312    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2psw\" (UniqueName: \"kubernetes.io/projected/345890db-ca81-4c74-a721-ba0904c50bf4-kube-api-access-p2psw\") pod \"multus-ppq4j\" (UID: \"345890db-ca81-4c74-a721-ba0904c50bf4\") " pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.294411 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.294313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6s2r\" (UniqueName: \"kubernetes.io/projected/7405cb77-3fa3-47ae-864d-b8b59f586932-kube-api-access-w6s2r\") pod \"multus-additional-cni-plugins-6jv4r\" (UID: \"7405cb77-3fa3-47ae-864d-b8b59f586932\") " pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.366588 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.366542    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6"
Jul 31 23:46:03.376673 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.376643    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pv7nj"
Jul 31 23:46:03.384496 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.384458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jxfkn"
Jul 31 23:46:03.392209 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.392185    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:03.398040 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.398020    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7ng6n"
Jul 31 23:46:03.404880 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.404859    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pzp69"
Jul 31 23:46:03.411657 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.411638    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5r99m"
Jul 31 23:46:03.418531 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.418513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6jv4r"
Jul 31 23:46:03.424582 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.424565    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ppq4j"
Jul 31 23:46:03.625842 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.625764    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:03.687459 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.687430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret\") pod \"global-pull-secret-syncer-zzsl6\" (UID: \"ab5415d7-3b45-4861-bfaf-be67c4d16871\") " pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:03.687659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.687488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs\") pod \"network-metrics-daemon-js2f5\" (UID: \"ec95cdf0-c411-409a-a0e5-274712ef6c00\") " pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:03.687659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:03.687524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pqmh\" (UniqueName: \"kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh\") pod \"network-check-target-mjkm4\" (UID: \"824b4cc2-7b16-43c6-9cc0-46759ffdf9b5\") " pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:03.687659 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.687578    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:03.687659 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.687638    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:03.687659 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.687647    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:03.687659 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.687664    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:03.687923 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.687647    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret podName:ab5415d7-3b45-4861-bfaf-be67c4d16871 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:04.68763107 +0000 UTC m=+4.047522090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret") pod "global-pull-secret-syncer-zzsl6" (UID: "ab5415d7-3b45-4861-bfaf-be67c4d16871") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:03.687923 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.687678    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9pqmh for pod openshift-network-diagnostics/network-check-target-mjkm4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:03.687923 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.687701    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs podName:ec95cdf0-c411-409a-a0e5-274712ef6c00 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:04.687681203 +0000 UTC m=+4.047572224 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs") pod "network-metrics-daemon-js2f5" (UID: "ec95cdf0-c411-409a-a0e5-274712ef6c00") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:03.687923 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:03.687749    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh podName:824b4cc2-7b16-43c6-9cc0-46759ffdf9b5 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:04.687733778 +0000 UTC m=+4.047624802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pqmh" (UniqueName: "kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh") pod "network-check-target-mjkm4" (UID: "824b4cc2-7b16-43c6-9cc0-46759ffdf9b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:04.063468 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:46:04.063440    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd94922_bfea_4c24_ac38_66daa163d31c.slice/crio-52b4e0ca488f8ef9e6820e36e7d090a7ef26ff426ffc1598510f32a9584a5dd5 WatchSource:0}: Error finding container 52b4e0ca488f8ef9e6820e36e7d090a7ef26ff426ffc1598510f32a9584a5dd5: Status 404 returned error can't find the container with id 52b4e0ca488f8ef9e6820e36e7d090a7ef26ff426ffc1598510f32a9584a5dd5
Jul 31 23:46:04.063997 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:46:04.063975    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7405cb77_3fa3_47ae_864d_b8b59f586932.slice/crio-3a6539be92d07fd7024fe6d03e30d431f5214b05aeb65f6ab67e1b8561ad8a0b WatchSource:0}: Error finding container 3a6539be92d07fd7024fe6d03e30d431f5214b05aeb65f6ab67e1b8561ad8a0b: Status 404 returned error can't find the container with id 3a6539be92d07fd7024fe6d03e30d431f5214b05aeb65f6ab67e1b8561ad8a0b
Jul 31 23:46:04.065101 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:46:04.065076    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345890db_ca81_4c74_a721_ba0904c50bf4.slice/crio-8163f146ebc02f2db83f6e28f8dd9e1b14ae24c0c41d9c1588252ba59afcae5a WatchSource:0}: Error finding container 8163f146ebc02f2db83f6e28f8dd9e1b14ae24c0c41d9c1588252ba59afcae5a: Status 404 returned error can't find the container with id 8163f146ebc02f2db83f6e28f8dd9e1b14ae24c0c41d9c1588252ba59afcae5a
Jul 31 23:46:04.066359 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:46:04.066336    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87b1f31_398c_45b4_a99c_5115633e0fc3.slice/crio-2d7eca172144ca1e2341dec78a024a0e054e8ef6aa5cf87ce0e18afcecd8e292 WatchSource:0}: Error finding container 2d7eca172144ca1e2341dec78a024a0e054e8ef6aa5cf87ce0e18afcecd8e292: Status 404 returned error can't find the container with id 2d7eca172144ca1e2341dec78a024a0e054e8ef6aa5cf87ce0e18afcecd8e292
Jul 31 23:46:04.067469 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:46:04.067440    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc459a9c2_d75c_44d9_865b_4262edeb4f61.slice/crio-30290ce58980a3075b0a5cce256144d1e6c231a2139486e1a1df8738adf4debd WatchSource:0}: Error finding container 30290ce58980a3075b0a5cce256144d1e6c231a2139486e1a1df8738adf4debd: Status 404 returned error can't find the container with id 30290ce58980a3075b0a5cce256144d1e6c231a2139486e1a1df8738adf4debd
Jul 31 23:46:04.068453 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:46:04.068424    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a3dcea_7f41_400b_b402_9cf780a6b030.slice/crio-c9dfc59e78ae083c6b5a998115031e3cd5bb5efc04dd25718b49ee17e43530d4 WatchSource:0}: Error finding container c9dfc59e78ae083c6b5a998115031e3cd5bb5efc04dd25718b49ee17e43530d4: Status 404 returned error can't find the container with id c9dfc59e78ae083c6b5a998115031e3cd5bb5efc04dd25718b49ee17e43530d4
Jul 31 23:46:04.076785 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:46:04.076764    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030ae37c_094e_4a10_bea2_8e2b8bcc6d5e.slice/crio-7843282dcde2e4115f86781d255b42c62820570d3335192a08eac588bbfb3b97 WatchSource:0}: Error finding container 7843282dcde2e4115f86781d255b42c62820570d3335192a08eac588bbfb3b97: Status 404 returned error can't find the container with id 7843282dcde2e4115f86781d255b42c62820570d3335192a08eac588bbfb3b97
Jul 31 23:46:04.079276 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:46:04.078392    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d1f0a2_5cd8_4cef_a805_f489af2e7931.slice/crio-942c3f290687321df0e1ba03dea61d50822e67fc73f3095bbfd212e295d8fa67 WatchSource:0}: Error finding container 942c3f290687321df0e1ba03dea61d50822e67fc73f3095bbfd212e295d8fa67: Status 404 returned error can't find the container with id 942c3f290687321df0e1ba03dea61d50822e67fc73f3095bbfd212e295d8fa67
Jul 31 23:46:04.108816 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:04.108766    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 23:41:03 +0000 UTC" deadline="2028-05-09 17:11:08.402285704 +0000 UTC"
Jul 31 23:46:04.108816 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:04.108811    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15545h25m4.293476864s"
Jul 31 23:46:04.179677 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:04.179475    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:04.179857 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:04.179799    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:04.195034 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:04.194995    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jv4r" event={"ID":"7405cb77-3fa3-47ae-864d-b8b59f586932","Type":"ContainerStarted","Data":"3a6539be92d07fd7024fe6d03e30d431f5214b05aeb65f6ab67e1b8561ad8a0b"}
Jul 31 23:46:04.195890 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:04.195865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5r99m" event={"ID":"d1a3dcea-7f41-400b-b402-9cf780a6b030","Type":"ContainerStarted","Data":"c9dfc59e78ae083c6b5a998115031e3cd5bb5efc04dd25718b49ee17e43530d4"}
Jul 31 23:46:04.199909 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:04.199884    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pv7nj" event={"ID":"a87b1f31-398c-45b4-a99c-5115633e0fc3","Type":"ContainerStarted","Data":"2d7eca172144ca1e2341dec78a024a0e054e8ef6aa5cf87ce0e18afcecd8e292"}
Jul 31 23:46:04.203400 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:04.203379    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6" event={"ID":"fdd94922-bfea-4c24-ac38-66daa163d31c","Type":"ContainerStarted","Data":"52b4e0ca488f8ef9e6820e36e7d090a7ef26ff426ffc1598510f32a9584a5dd5"}
Jul 31 23:46:04.204364 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:04.204338    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pzp69" event={"ID":"42d1f0a2-5cd8-4cef-a805-f489af2e7931","Type":"ContainerStarted","Data":"942c3f290687321df0e1ba03dea61d50822e67fc73f3095bbfd212e295d8fa67"}
Jul 31 23:46:04.205462 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:04.205436    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq" event={"ID":"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e","Type":"ContainerStarted","Data":"7843282dcde2e4115f86781d255b42c62820570d3335192a08eac588bbfb3b97"}
Jul 31 23:46:04.209430 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:04.209411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jxfkn" event={"ID":"c459a9c2-d75c-44d9-865b-4262edeb4f61","Type":"ContainerStarted","Data":"30290ce58980a3075b0a5cce256144d1e6c231a2139486e1a1df8738adf4debd"}
Jul 31 23:46:04.212418 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:04.212399    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ppq4j" event={"ID":"345890db-ca81-4c74-a721-ba0904c50bf4","Type":"ContainerStarted","Data":"8163f146ebc02f2db83f6e28f8dd9e1b14ae24c0c41d9c1588252ba59afcae5a"}
Jul 31 23:46:04.214850 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:04.214828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal" event={"ID":"8a89e13fce40802bf99f22858790a870","Type":"ContainerStarted","Data":"c255fd39c0bda5674f141e4ba732f3eb88b1072ed4fb21a1093246f08d09579f"}
Jul 31 23:46:04.216650 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:04.216625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7ng6n" event={"ID":"1ddbfa22-7b8f-4800-8dd1-78a871bbf08d","Type":"ContainerStarted","Data":"7422dc4afa17cc471fe3ffe603ce9d02c0e7b6e43463cdb264772260dd5011ee"}
Jul 31 23:46:04.227358 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:04.227309    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal" podStartSLOduration=2.22729289 podStartE2EDuration="2.22729289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:04.22696156 +0000 UTC m=+3.586852600" watchObservedRunningTime="2026-07-31 23:46:04.22729289 +0000 UTC m=+3.587183948"
Jul 31 23:46:04.694803 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:04.694743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs\") pod \"network-metrics-daemon-js2f5\" (UID: \"ec95cdf0-c411-409a-a0e5-274712ef6c00\") " pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:04.694803 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:04.694808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pqmh\" (UniqueName: \"kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh\") pod \"network-check-target-mjkm4\" (UID: \"824b4cc2-7b16-43c6-9cc0-46759ffdf9b5\") " pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:04.695046 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:04.694855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret\") pod \"global-pull-secret-syncer-zzsl6\" (UID: \"ab5415d7-3b45-4861-bfaf-be67c4d16871\") " pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:04.695046 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:04.694990    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:04.695148 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:04.695056    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret podName:ab5415d7-3b45-4861-bfaf-be67c4d16871 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:06.695037039 +0000 UTC m=+6.054928078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret") pod "global-pull-secret-syncer-zzsl6" (UID: "ab5415d7-3b45-4861-bfaf-be67c4d16871") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:04.695494 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:04.695474    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:04.695559 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:04.695540    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs podName:ec95cdf0-c411-409a-a0e5-274712ef6c00 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:06.6955209 +0000 UTC m=+6.055411925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs") pod "network-metrics-daemon-js2f5" (UID: "ec95cdf0-c411-409a-a0e5-274712ef6c00") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:04.695644 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:04.695631    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:04.695705 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:04.695650    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:04.695705 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:04.695663    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9pqmh for pod openshift-network-diagnostics/network-check-target-mjkm4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:04.695705 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:04.695702    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh podName:824b4cc2-7b16-43c6-9cc0-46759ffdf9b5 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:06.69568952 +0000 UTC m=+6.055580545 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pqmh" (UniqueName: "kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh") pod "network-check-target-mjkm4" (UID: "824b4cc2-7b16-43c6-9cc0-46759ffdf9b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:05.109690 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:05.109594    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 23:41:03 +0000 UTC" deadline="2028-05-18 22:05:12.098914664 +0000 UTC"
Jul 31 23:46:05.109690 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:05.109633    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15766h19m6.989287339s"
Jul 31 23:46:05.183216 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:05.181869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:05.183216 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:05.182039    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:05.183216 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:05.182533    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:05.183216 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:05.182628    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:05.231403 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:05.231367    2578 generic.go:356] "Generic (PLEG): container finished" podID="d96382247de37d9245ec69df207c39f4" containerID="1a41d037eea2358cd3abd23348c633fb4e4e34cae4de7dfc7a32db6f2b801213" exitCode=0
Jul 31 23:46:05.231575 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:05.231462    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal" event={"ID":"d96382247de37d9245ec69df207c39f4","Type":"ContainerDied","Data":"1a41d037eea2358cd3abd23348c633fb4e4e34cae4de7dfc7a32db6f2b801213"}
Jul 31 23:46:06.179481 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:06.179445    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:06.179979 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:06.179572    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:06.240784 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:06.240630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal" event={"ID":"d96382247de37d9245ec69df207c39f4","Type":"ContainerStarted","Data":"2ef30324e40961cda10c92060ed905e14fbcc40286dcd25ef5dab04e64400b3a"}
Jul 31 23:46:06.260253 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:06.259672    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal" podStartSLOduration=4.259649138 podStartE2EDuration="4.259649138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:06.257937046 +0000 UTC m=+5.617828114" watchObservedRunningTime="2026-07-31 23:46:06.259649138 +0000 UTC m=+5.619540184"
Jul 31 23:46:06.713190 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:06.713151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs\") pod \"network-metrics-daemon-js2f5\" (UID: \"ec95cdf0-c411-409a-a0e5-274712ef6c00\") " pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:06.713359 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:06.713221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pqmh\" (UniqueName: \"kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh\") pod \"network-check-target-mjkm4\" (UID: \"824b4cc2-7b16-43c6-9cc0-46759ffdf9b5\") " pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:06.713359 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:06.713261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret\") pod \"global-pull-secret-syncer-zzsl6\" (UID: \"ab5415d7-3b45-4861-bfaf-be67c4d16871\") " pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:06.713481 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:06.713400    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:06.713481 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:06.713463    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret podName:ab5415d7-3b45-4861-bfaf-be67c4d16871 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:10.713445005 +0000 UTC m=+10.073336040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret") pod "global-pull-secret-syncer-zzsl6" (UID: "ab5415d7-3b45-4861-bfaf-be67c4d16871") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:06.714385 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:06.713884    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:06.714385 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:06.713945    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs podName:ec95cdf0-c411-409a-a0e5-274712ef6c00 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:10.713928176 +0000 UTC m=+10.073819264 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs") pod "network-metrics-daemon-js2f5" (UID: "ec95cdf0-c411-409a-a0e5-274712ef6c00") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:06.714385 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:06.714030    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:06.714385 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:06.714045    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:06.714385 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:06.714058    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9pqmh for pod openshift-network-diagnostics/network-check-target-mjkm4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:06.714385 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:06.714098    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh podName:824b4cc2-7b16-43c6-9cc0-46759ffdf9b5 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:10.714084853 +0000 UTC m=+10.073975885 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pqmh" (UniqueName: "kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh") pod "network-check-target-mjkm4" (UID: "824b4cc2-7b16-43c6-9cc0-46759ffdf9b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:07.182097 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:07.182010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:07.182534 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:07.182148    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:07.182607 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:07.182594    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:07.183058 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:07.182978    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:08.179449 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:08.179415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:08.179631 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:08.179542    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:09.182515 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:09.181827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:09.182515 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:09.181956    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:09.182515 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:09.182330    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:09.182515 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:09.182415    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:10.180394 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:10.179873    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:10.180394 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:10.180007    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:10.748907 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:10.748862    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret\") pod \"global-pull-secret-syncer-zzsl6\" (UID: \"ab5415d7-3b45-4861-bfaf-be67c4d16871\") " pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:10.749381 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:10.748986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs\") pod \"network-metrics-daemon-js2f5\" (UID: \"ec95cdf0-c411-409a-a0e5-274712ef6c00\") " pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:10.749381 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:10.749029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pqmh\" (UniqueName: \"kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh\") pod \"network-check-target-mjkm4\" (UID: \"824b4cc2-7b16-43c6-9cc0-46759ffdf9b5\") " pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:10.749381 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:10.749307    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:10.749381 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:10.749329    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:10.749381 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:10.749341    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9pqmh for pod openshift-network-diagnostics/network-check-target-mjkm4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:10.749645 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:10.749402    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh podName:824b4cc2-7b16-43c6-9cc0-46759ffdf9b5 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:18.749382815 +0000 UTC m=+18.109273851 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pqmh" (UniqueName: "kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh") pod "network-check-target-mjkm4" (UID: "824b4cc2-7b16-43c6-9cc0-46759ffdf9b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:10.749888 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:10.749869    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:10.749964 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:10.749926    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret podName:ab5415d7-3b45-4861-bfaf-be67c4d16871 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:18.749910502 +0000 UTC m=+18.109801528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret") pod "global-pull-secret-syncer-zzsl6" (UID: "ab5415d7-3b45-4861-bfaf-be67c4d16871") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:10.750031 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:10.749985    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:10.750031 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:10.750016    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs podName:ec95cdf0-c411-409a-a0e5-274712ef6c00 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:18.750005408 +0000 UTC m=+18.109896432 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs") pod "network-metrics-daemon-js2f5" (UID: "ec95cdf0-c411-409a-a0e5-274712ef6c00") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:11.180132 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:11.180063    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:11.180266 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:11.180157    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:11.180575 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:11.180556    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:11.180684 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:11.180662    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:12.179882 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:12.179362    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:12.179882 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:12.179498    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:13.180071 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:13.179288    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:13.180071 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:13.179530    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:13.180071 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:13.179924    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:13.180071 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:13.180023    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:14.179144 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:14.179102    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:14.179347 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:14.179226    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:15.179670 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:15.179634    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:15.180143 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:15.179684    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:15.180143 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:15.179795    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:15.180143 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:15.179905    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:16.179730 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:16.179690    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:16.180112 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:16.179846    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:17.179245 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:17.179209    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:17.179426 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:17.179342    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:17.179426 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:17.179400    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:17.179538 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:17.179495    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:18.179554 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:18.179518    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:18.180041 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:18.179647    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:18.813086 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:18.813054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs\") pod \"network-metrics-daemon-js2f5\" (UID: \"ec95cdf0-c411-409a-a0e5-274712ef6c00\") " pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:18.813199 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:18.813106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pqmh\" (UniqueName: \"kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh\") pod \"network-check-target-mjkm4\" (UID: \"824b4cc2-7b16-43c6-9cc0-46759ffdf9b5\") " pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:18.813199 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:18.813156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret\") pod \"global-pull-secret-syncer-zzsl6\" (UID: \"ab5415d7-3b45-4861-bfaf-be67c4d16871\") " pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:18.813278 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:18.813198    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:18.813278 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:18.813244    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:18.813278 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:18.813261    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:18.813278 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:18.813269    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:18.813441 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:18.813285    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9pqmh for pod openshift-network-diagnostics/network-check-target-mjkm4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:18.813441 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:18.813263    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs podName:ec95cdf0-c411-409a-a0e5-274712ef6c00 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:34.813246913 +0000 UTC m=+34.173137935 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs") pod "network-metrics-daemon-js2f5" (UID: "ec95cdf0-c411-409a-a0e5-274712ef6c00") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:18.813441 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:18.813342    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret podName:ab5415d7-3b45-4861-bfaf-be67c4d16871 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:34.813327449 +0000 UTC m=+34.173218472 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret") pod "global-pull-secret-syncer-zzsl6" (UID: "ab5415d7-3b45-4861-bfaf-be67c4d16871") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:18.813441 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:18.813354    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh podName:824b4cc2-7b16-43c6-9cc0-46759ffdf9b5 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:34.813348654 +0000 UTC m=+34.173239673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pqmh" (UniqueName: "kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh") pod "network-check-target-mjkm4" (UID: "824b4cc2-7b16-43c6-9cc0-46759ffdf9b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:19.179939 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:19.179373    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:19.179939 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:19.179404    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:19.179939 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:19.179518    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:19.179939 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:19.179664    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:20.179368 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:20.179340    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:20.179565 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:20.179453    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:21.180688 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:21.180650    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:21.181113 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:21.180790    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:21.181113 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:21.180854    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:21.181113 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:21.180967    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:22.179776 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:22.179738    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:22.179940 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:22.179863    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:23.179662 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:23.179470    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:23.180312 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:23.179503    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:23.180312 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:23.179802    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:23.180312 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:23.179906    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:23.275505 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:23.275472    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq" event={"ID":"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e","Type":"ContainerStarted","Data":"d590826d8746b5669212c6ea39abaa747f3f553187c1d3ab3cbc6f0bfc756b9e"}
Jul 31 23:46:23.277282 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:23.277257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jxfkn" event={"ID":"c459a9c2-d75c-44d9-865b-4262edeb4f61","Type":"ContainerStarted","Data":"81cb3e742283197ea65d033ea30a788b8c3d7687b39e779cc106a114cad2ef62"}
Jul 31 23:46:23.279691 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:23.279661    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ppq4j" event={"ID":"345890db-ca81-4c74-a721-ba0904c50bf4","Type":"ContainerStarted","Data":"da2bb6d695df34c8ffc2a80f40a661b53376533fe65af892a349e8a12ef057a2"}
Jul 31 23:46:23.281506 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:23.281481    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7ng6n" event={"ID":"1ddbfa22-7b8f-4800-8dd1-78a871bbf08d","Type":"ContainerStarted","Data":"4b83566ce085dc8ceaee9bf42b1b300b8c6eeb6d178ea00f3ecb961bc17cf512"}
Jul 31 23:46:23.283302 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:23.283278    2578 generic.go:356] "Generic (PLEG): container finished" podID="7405cb77-3fa3-47ae-864d-b8b59f586932" containerID="d6e8e16d44607f51a2cd329e6a42d4ebda111b39c36fc6448b1d4963a612ec77" exitCode=0
Jul 31 23:46:23.283400 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:23.283359    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jv4r" event={"ID":"7405cb77-3fa3-47ae-864d-b8b59f586932","Type":"ContainerDied","Data":"d6e8e16d44607f51a2cd329e6a42d4ebda111b39c36fc6448b1d4963a612ec77"}
Jul 31 23:46:23.285054 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:23.285032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5r99m" event={"ID":"d1a3dcea-7f41-400b-b402-9cf780a6b030","Type":"ContainerStarted","Data":"f5652738612f15887d1dfa59331421978563a299eedc9826dea745cfae513be5"}
Jul 31 23:46:23.286862 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:23.286839    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pv7nj" event={"ID":"a87b1f31-398c-45b4-a99c-5115633e0fc3","Type":"ContainerStarted","Data":"bb6239f0b725e4546cffc1f164340ea689040afcf01f0557cafe9ea5d9604c9c"}
Jul 31 23:46:23.288389 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:23.288365    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6" event={"ID":"fdd94922-bfea-4c24-ac38-66daa163d31c","Type":"ContainerStarted","Data":"564c227cf1b986a2fd5707d65bfac919f36a416485535daed2c173b08b70d03c"}
Jul 31 23:46:23.296747 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:23.296686    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jxfkn" podStartSLOduration=3.549486017 podStartE2EDuration="22.296663935s" totalImagesPullingTime="18.747177918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:23.29647257 +0000 UTC m=+22.656363611" watchObservedRunningTime="2026-07-31 23:46:23.296663935 +0000 UTC m=+22.656554977"
Jul 31 23:46:23.315527 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:23.315488    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5r99m" podStartSLOduration=3.56835105 podStartE2EDuration="22.315473003s" totalImagesPullingTime="18.747121953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:23.315066322 +0000 UTC m=+22.674957363" watchObservedRunningTime="2026-07-31 23:46:23.315473003 +0000 UTC m=+22.675364044"
Jul 31 23:46:23.373348 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:23.373303    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ppq4j" podStartSLOduration=3.585248873 podStartE2EDuration="22.373281321s" totalImagesPullingTime="18.788032448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:23.355479081 +0000 UTC m=+22.715370133" watchObservedRunningTime="2026-07-31 23:46:23.373281321 +0000 UTC m=+22.733172363"
Jul 31 23:46:23.373745 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:23.373696    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pv7nj" podStartSLOduration=3.6232392239999998 podStartE2EDuration="22.373689226s" totalImagesPullingTime="18.750450002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:23.373387207 +0000 UTC m=+22.733278249" watchObservedRunningTime="2026-07-31 23:46:23.373689226 +0000 UTC m=+22.733580268"
Jul 31 23:46:23.389311 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:23.389275    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7ng6n" podStartSLOduration=7.696726179 podStartE2EDuration="22.3892597s" totalImagesPullingTime="14.692533521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:23.388691315 +0000 UTC m=+22.748582355" watchObservedRunningTime="2026-07-31 23:46:23.3892597 +0000 UTC m=+22.749150741"
Jul 31 23:46:24.179267 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:24.179028    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:24.179425 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:24.179357    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:24.294014 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:24.293967    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Jul 31 23:46:24.294530 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:24.294424    2578 generic.go:356] "Generic (PLEG): container finished" podID="030ae37c-094e-4a10-bea2-8e2b8bcc6d5e" containerID="4074fb2c3819b653edbcbd90bd4240fc8cf89d4b990ee004a81b34953e457e43" exitCode=1
Jul 31 23:46:24.294998 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:24.294973    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq" event={"ID":"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e","Type":"ContainerStarted","Data":"4823a2038b29be6e3681bfa0f8c093e0ec03846bbc5f7f6dc556c3781a682db5"}
Jul 31 23:46:24.295102 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:24.295059    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq" event={"ID":"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e","Type":"ContainerStarted","Data":"38224f0b64f1d50670b90c5715b2cfe3ca6e30888722e8468b567678c2e63041"}
Jul 31 23:46:24.295102 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:24.295092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq" event={"ID":"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e","Type":"ContainerStarted","Data":"360b3351a50f5852971b399aa996ea9c3700aaf232b4f53d58bf4ef4b8255133"}
Jul 31 23:46:24.295214 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:24.295135    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq" event={"ID":"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e","Type":"ContainerStarted","Data":"6be7e69b986ff9ba09e1d3d57dc09f061ea390c1643087654b98b9e73e8726ec"}
Jul 31 23:46:24.295214 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:24.295180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq" event={"ID":"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e","Type":"ContainerDied","Data":"4074fb2c3819b653edbcbd90bd4240fc8cf89d4b990ee004a81b34953e457e43"}
Jul 31 23:46:24.404546 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:24.404160    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 31 23:46:25.134846 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:25.134741    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T23:46:24.404174451Z","UUID":"88b18382-a8ab-4c44-9751-446006be7925","Handler":null,"Name":"","Endpoint":""}
Jul 31 23:46:25.136705 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:25.136672    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 31 23:46:25.136926 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:25.136735    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 31 23:46:25.179626 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:25.179584    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:25.179812 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:25.179746    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:25.179812 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:25.179778    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:25.179919 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:25.179854    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:25.214985 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:25.214957    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7ng6n"
Jul 31 23:46:25.216084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:25.215876    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7ng6n"
Jul 31 23:46:25.299172 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:25.299135    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6" event={"ID":"fdd94922-bfea-4c24-ac38-66daa163d31c","Type":"ContainerStarted","Data":"66340cd846593e1a7174dc0ef9f51be6029a263fc858c47a3af09a0890b782ad"}
Jul 31 23:46:25.301880 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:25.301828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pzp69" event={"ID":"42d1f0a2-5cd8-4cef-a805-f489af2e7931","Type":"ContainerStarted","Data":"051b90d8ed611c5ebd336922b4e9279da7a5abac3819326048961292ee8f88b9"}
Jul 31 23:46:25.302028 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:25.301931    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7ng6n"
Jul 31 23:46:25.302365 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:25.302345    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7ng6n"
Jul 31 23:46:25.316284 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:25.316231    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pzp69" podStartSLOduration=5.572723869 podStartE2EDuration="24.316210631s" totalImagesPullingTime="18.743486762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:25.315566719 +0000 UTC m=+24.675457791" watchObservedRunningTime="2026-07-31 23:46:25.316210631 +0000 UTC m=+24.676101675"
Jul 31 23:46:26.179153 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:26.179120    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:26.179346 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:26.179227    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:26.305782 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:26.305744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6" event={"ID":"fdd94922-bfea-4c24-ac38-66daa163d31c","Type":"ContainerStarted","Data":"cd0ccf3b989309474e7155e85b9491211380dd087e7c9f4fd9b41de26d6b5462"}
Jul 31 23:46:26.323675 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:26.323626    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgtw6" podStartSLOduration=4.29300625 podStartE2EDuration="25.323611844s" totalImagesPullingTime="21.030605594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:26.32328355 +0000 UTC m=+25.683174591" watchObservedRunningTime="2026-07-31 23:46:26.323611844 +0000 UTC m=+25.683502863"
Jul 31 23:46:27.180041 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:27.179861    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:27.180205 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:27.179869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:27.180205 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:27.180153    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:27.180362 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:27.180246    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:27.310682 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:27.310657    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Jul 31 23:46:27.311124 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:27.311098    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq" event={"ID":"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e","Type":"ContainerStarted","Data":"cb901af1c32285184e8ed4d99933a4d3f0d65d298e714e15e682bdb6dc449ef5"}
Jul 31 23:46:27.737822 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:27.737794    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5r99m_d1a3dcea-7f41-400b-b402-9cf780a6b030/dns-node-resolver/0.log"
Jul 31 23:46:28.179734 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:28.179620    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:28.179866 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:28.179755    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:28.315086 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:28.315054    2578 generic.go:356] "Generic (PLEG): container finished" podID="7405cb77-3fa3-47ae-864d-b8b59f586932" containerID="3a67e5abb07b0ee5e5c8cdb99a35fd707a2bf4059166f2c3999761c35b75b307" exitCode=0
Jul 31 23:46:28.315543 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:28.315101    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jv4r" event={"ID":"7405cb77-3fa3-47ae-864d-b8b59f586932","Type":"ContainerDied","Data":"3a67e5abb07b0ee5e5c8cdb99a35fd707a2bf4059166f2c3999761c35b75b307"}
Jul 31 23:46:28.719296 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:28.719266    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jxfkn_c459a9c2-d75c-44d9-865b-4262edeb4f61/node-ca/0.log"
Jul 31 23:46:29.180049 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:29.180017    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:29.180213 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:29.180060    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:29.180213 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:29.180143    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:29.180310 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:29.180270    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:29.319469 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:29.319445    2578 generic.go:356] "Generic (PLEG): container finished" podID="7405cb77-3fa3-47ae-864d-b8b59f586932" containerID="02d8d793c249b17ffc427f60d901c0a357d93d77a27cf441c8ea85e34bca3b87" exitCode=0
Jul 31 23:46:29.319872 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:29.319485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jv4r" event={"ID":"7405cb77-3fa3-47ae-864d-b8b59f586932","Type":"ContainerDied","Data":"02d8d793c249b17ffc427f60d901c0a357d93d77a27cf441c8ea85e34bca3b87"}
Jul 31 23:46:30.180196 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:30.179937    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:30.180333 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:30.180223    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:30.324645 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:30.324618    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Jul 31 23:46:30.325047 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:30.324988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq" event={"ID":"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e","Type":"ContainerStarted","Data":"337bc802a706e9bff6d95de6912fa30cee87ff38155d7b64ac87b9fc4d391c49"}
Jul 31 23:46:30.325264 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:30.325242    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:30.325386 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:30.325307    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:30.325453 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:30.325410    2578 scope.go:117] "RemoveContainer" containerID="4074fb2c3819b653edbcbd90bd4240fc8cf89d4b990ee004a81b34953e457e43"
Jul 31 23:46:30.327531 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:30.327502    2578 generic.go:356] "Generic (PLEG): container finished" podID="7405cb77-3fa3-47ae-864d-b8b59f586932" containerID="1628fa33d63e98a2055def3e89b3deb72952c5aa1ebfa2e59f6c8e6bf38b603e" exitCode=0
Jul 31 23:46:30.327625 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:30.327541    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jv4r" event={"ID":"7405cb77-3fa3-47ae-864d-b8b59f586932","Type":"ContainerDied","Data":"1628fa33d63e98a2055def3e89b3deb72952c5aa1ebfa2e59f6c8e6bf38b603e"}
Jul 31 23:46:30.343045 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:30.343018    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:31.181378 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:31.181166    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:31.181508 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:31.181482    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:31.181700 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:31.181223    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:31.181700 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:31.181664    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:31.325938 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:31.325903    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mjkm4"]
Jul 31 23:46:31.326654 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:31.326625    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zzsl6"]
Jul 31 23:46:31.326790 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:31.326767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:31.326891 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:31.326868    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:31.330938 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:31.327979    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-js2f5"]
Jul 31 23:46:31.338283 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:31.338224    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Jul 31 23:46:31.338664 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:31.338621    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq" event={"ID":"030ae37c-094e-4a10-bea2-8e2b8bcc6d5e","Type":"ContainerStarted","Data":"7def9bb080809c19f3e6b3450281152e07b774610e7f76abfa20b9d5f9537072"}
Jul 31 23:46:31.338785 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:31.338666    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:31.338785 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:31.338649    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:31.338867 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:31.338802    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:31.339004 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:31.338979    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:31.339154 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:31.339121    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:31.360012 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:31.359981    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:46:31.375602 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:31.375539    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq" podStartSLOduration=11.384205946 podStartE2EDuration="30.375515519s" totalImagesPullingTime="18.991309573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:31.371941442 +0000 UTC m=+30.731832506" watchObservedRunningTime="2026-07-31 23:46:31.375515519 +0000 UTC m=+30.735406565"
Jul 31 23:46:33.188416 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:33.188388    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:33.189001 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:33.188468    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:33.189001 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:33.188492    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:33.189001 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:33.188503    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:33.189001 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:33.188602    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:33.189001 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:33.188692    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:34.838915 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:34.838881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs\") pod \"network-metrics-daemon-js2f5\" (UID: \"ec95cdf0-c411-409a-a0e5-274712ef6c00\") " pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:34.838915 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:34.838922    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pqmh\" (UniqueName: \"kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh\") pod \"network-check-target-mjkm4\" (UID: \"824b4cc2-7b16-43c6-9cc0-46759ffdf9b5\") " pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:34.839529 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:34.838948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret\") pod \"global-pull-secret-syncer-zzsl6\" (UID: \"ab5415d7-3b45-4861-bfaf-be67c4d16871\") " pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:34.839529 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:34.839025    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:34.839529 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:34.839026    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:34.839529 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:34.839046    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:34.839529 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:34.839062    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:34.839529 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:34.839072    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9pqmh for pod openshift-network-diagnostics/network-check-target-mjkm4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:34.839529 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:34.839076    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret podName:ab5415d7-3b45-4861-bfaf-be67c4d16871 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:06.839063692 +0000 UTC m=+66.198954711 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret") pod "global-pull-secret-syncer-zzsl6" (UID: "ab5415d7-3b45-4861-bfaf-be67c4d16871") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:34.839529 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:34.839089    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs podName:ec95cdf0-c411-409a-a0e5-274712ef6c00 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:06.839082511 +0000 UTC m=+66.198973530 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs") pod "network-metrics-daemon-js2f5" (UID: "ec95cdf0-c411-409a-a0e5-274712ef6c00") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:34.839529 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:34.839112    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh podName:824b4cc2-7b16-43c6-9cc0-46759ffdf9b5 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:06.839096228 +0000 UTC m=+66.198987250 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pqmh" (UniqueName: "kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh") pod "network-check-target-mjkm4" (UID: "824b4cc2-7b16-43c6-9cc0-46759ffdf9b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:35.179625 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:35.179584    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:35.179825 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:35.179624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:35.179825 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:35.179710    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:35.179920 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:35.179815    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:35.179920 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:35.179872    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:35.179992 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:35.179957    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:35.350376 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:35.350346    2578 generic.go:356] "Generic (PLEG): container finished" podID="7405cb77-3fa3-47ae-864d-b8b59f586932" containerID="503b8224ea795f0098faac9cd7b9fdf0a9a374f488541ef9837337c108cb4a61" exitCode=0
Jul 31 23:46:35.350531 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:35.350385    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jv4r" event={"ID":"7405cb77-3fa3-47ae-864d-b8b59f586932","Type":"ContainerDied","Data":"503b8224ea795f0098faac9cd7b9fdf0a9a374f488541ef9837337c108cb4a61"}
Jul 31 23:46:36.355382 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:36.355348    2578 generic.go:356] "Generic (PLEG): container finished" podID="7405cb77-3fa3-47ae-864d-b8b59f586932" containerID="858f4bc937e91081f8dca200bf89e874b83347e57808132418ec665018620c35" exitCode=0
Jul 31 23:46:36.355893 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:36.355427    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jv4r" event={"ID":"7405cb77-3fa3-47ae-864d-b8b59f586932","Type":"ContainerDied","Data":"858f4bc937e91081f8dca200bf89e874b83347e57808132418ec665018620c35"}
Jul 31 23:46:37.182579 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:37.182545    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:37.182579 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:37.182569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:37.182789 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:37.182666    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:37.182789 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:37.182727    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:37.182854 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:37.182822    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:37.182932 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:37.182916    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:37.360874 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:37.360839    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jv4r" event={"ID":"7405cb77-3fa3-47ae-864d-b8b59f586932","Type":"ContainerStarted","Data":"f11c92eb7e68d1fbb13340a704888c8ee5c2b06f60579b06ded9018b9e587a99"}
Jul 31 23:46:39.179793 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:39.179579    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:39.180258 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:39.179587    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:39.180258 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:39.179886    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:39.180258 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:39.179615    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:39.180258 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:39.179964    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:39.180258 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:39.180026    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:41.179757 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:41.179705    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:41.180264 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:41.179796    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:41.180264 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:41.179811    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:41.180264 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:41.179901    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:41.180264 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:41.179927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:41.180264 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:41.179981    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:43.179786 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:43.179747    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:43.179786 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:43.179765    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:43.180276 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:43.179871    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:43.180276 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:43.179902    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:43.180276 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:43.179962    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:43.180276 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:43.180009    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:45.182730 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:45.182693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:45.183113 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:45.182746    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:45.183113 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:45.182872    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:45.183113 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:45.182906    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:45.183113 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:45.182952    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:45.183113 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:45.183044    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:47.179648 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:47.179616    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:47.180097 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:47.179620    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:47.180097 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:47.179750    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:47.180097 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:47.179621    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:47.180097 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:47.179819    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:47.180097 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:47.179926    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:49.182328 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:49.182295    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:49.182687 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:49.182381    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:49.182687 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:49.182387    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:49.182687 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:49.182407    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:49.182687 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:49.182485    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:49.182687 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:49.182564    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:51.180284 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:51.180248    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:51.180871 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:51.180366    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:51.180871 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:51.180374    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:51.180871 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:51.180475    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:51.180871 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:51.180516    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:51.180871 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:51.180552    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:53.179308 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:53.179266    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:53.179819 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:53.179266    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:53.179819 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:53.179379    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-js2f5" podUID="ec95cdf0-c411-409a-a0e5-274712ef6c00"
Jul 31 23:46:53.179819 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:53.179402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:53.179819 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:53.179484    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjkm4" podUID="824b4cc2-7b16-43c6-9cc0-46759ffdf9b5"
Jul 31 23:46:53.179819 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:53.179559    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzsl6" podUID="ab5415d7-3b45-4861-bfaf-be67c4d16871"
Jul 31 23:46:53.991139 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:53.991109    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeReady"
Jul 31 23:46:53.991328 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:53.991243    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 31 23:46:54.050184 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.050093    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6jv4r" podStartSLOduration=24.452025325 podStartE2EDuration="53.050078511s" totalImagesPullingTime="28.598053186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:37.386351994 +0000 UTC m=+36.746243034" watchObservedRunningTime="2026-07-31 23:46:54.050078511 +0000 UTC m=+53.409969548"
Jul 31 23:46:54.050594 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.050511    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5nwm2"]
Jul 31 23:46:54.074819 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.074797    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gvwvq"]
Jul 31 23:46:54.074956 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.074932    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5nwm2"
Jul 31 23:46:54.077156 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.077133    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tss5j\""
Jul 31 23:46:54.078960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.078940    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 31 23:46:54.078960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.078944    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 31 23:46:54.094754 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.094733    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5nwm2"]
Jul 31 23:46:54.094851 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.094796    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gvwvq"
Jul 31 23:46:54.098312 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.098296    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 31 23:46:54.101925 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.101906    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gvwvq"]
Jul 31 23:46:54.124940 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.124920    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 31 23:46:54.125111 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.125092    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 31 23:46:54.125498 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.125484    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qx4sg\""
Jul 31 23:46:54.139360 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.139338    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5qwfc"]
Jul 31 23:46:54.157520 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.157491    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5qwfc"
Jul 31 23:46:54.161656 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.161635    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 31 23:46:54.161861 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.161765    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 31 23:46:54.161951 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.161895    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 31 23:46:54.161951 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.161896    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 31 23:46:54.162044 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.161954    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mw9xk\""
Jul 31 23:46:54.165290 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.165272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1de6b043-026d-4c14-a302-25c6357dc375-tmp-dir\") pod \"dns-default-5nwm2\" (UID: \"1de6b043-026d-4c14-a302-25c6357dc375\") " pod="openshift-dns/dns-default-5nwm2"
Jul 31 23:46:54.165366 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.165307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1de6b043-026d-4c14-a302-25c6357dc375-metrics-tls\") pod \"dns-default-5nwm2\" (UID: \"1de6b043-026d-4c14-a302-25c6357dc375\") " pod="openshift-dns/dns-default-5nwm2"
Jul 31 23:46:54.165366 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.165324    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsbjp\" (UniqueName: \"kubernetes.io/projected/93a82855-64d9-4eb2-bdc5-fb4cfef63f88-kube-api-access-zsbjp\") pod \"ingress-canary-gvwvq\" (UID: \"93a82855-64d9-4eb2-bdc5-fb4cfef63f88\") " pod="openshift-ingress-canary/ingress-canary-gvwvq"
Jul 31 23:46:54.165451 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.165423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbvlx\" (UniqueName: \"kubernetes.io/projected/1de6b043-026d-4c14-a302-25c6357dc375-kube-api-access-nbvlx\") pod \"dns-default-5nwm2\" (UID: \"1de6b043-026d-4c14-a302-25c6357dc375\") " pod="openshift-dns/dns-default-5nwm2"
Jul 31 23:46:54.165484 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.165450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93a82855-64d9-4eb2-bdc5-fb4cfef63f88-cert\") pod \"ingress-canary-gvwvq\" (UID: \"93a82855-64d9-4eb2-bdc5-fb4cfef63f88\") " pod="openshift-ingress-canary/ingress-canary-gvwvq"
Jul 31 23:46:54.165484 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.165476    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1de6b043-026d-4c14-a302-25c6357dc375-config-volume\") pod \"dns-default-5nwm2\" (UID: \"1de6b043-026d-4c14-a302-25c6357dc375\") " pod="openshift-dns/dns-default-5nwm2"
Jul 31 23:46:54.183512 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.183482    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5qwfc"]
Jul 31 23:46:54.265873 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.265787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93a82855-64d9-4eb2-bdc5-fb4cfef63f88-cert\") pod \"ingress-canary-gvwvq\" (UID: \"93a82855-64d9-4eb2-bdc5-fb4cfef63f88\") " pod="openshift-ingress-canary/ingress-canary-gvwvq"
Jul 31 23:46:54.265873 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.265827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f1719f9-a204-4ba5-bf61-1197626af667-crio-socket\") pod \"insights-runtime-extractor-5qwfc\" (UID: \"7f1719f9-a204-4ba5-bf61-1197626af667\") " pod="openshift-insights/insights-runtime-extractor-5qwfc"
Jul 31 23:46:54.265873 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.265856    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1de6b043-026d-4c14-a302-25c6357dc375-config-volume\") pod \"dns-default-5nwm2\" (UID: \"1de6b043-026d-4c14-a302-25c6357dc375\") " pod="openshift-dns/dns-default-5nwm2"
Jul 31 23:46:54.266114 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.265932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f1719f9-a204-4ba5-bf61-1197626af667-data-volume\") pod \"insights-runtime-extractor-5qwfc\" (UID: \"7f1719f9-a204-4ba5-bf61-1197626af667\") " pod="openshift-insights/insights-runtime-extractor-5qwfc"
Jul 31 23:46:54.266114 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.265972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1de6b043-026d-4c14-a302-25c6357dc375-tmp-dir\") pod \"dns-default-5nwm2\" (UID: \"1de6b043-026d-4c14-a302-25c6357dc375\") " pod="openshift-dns/dns-default-5nwm2"
Jul 31 23:46:54.266114 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.266010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1de6b043-026d-4c14-a302-25c6357dc375-metrics-tls\") pod \"dns-default-5nwm2\" (UID: \"1de6b043-026d-4c14-a302-25c6357dc375\") " pod="openshift-dns/dns-default-5nwm2"
Jul 31 23:46:54.266114 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.266030    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsbjp\" (UniqueName: \"kubernetes.io/projected/93a82855-64d9-4eb2-bdc5-fb4cfef63f88-kube-api-access-zsbjp\") pod \"ingress-canary-gvwvq\" (UID: \"93a82855-64d9-4eb2-bdc5-fb4cfef63f88\") " pod="openshift-ingress-canary/ingress-canary-gvwvq"
Jul 31 23:46:54.266114 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.266057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f1719f9-a204-4ba5-bf61-1197626af667-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qwfc\" (UID: \"7f1719f9-a204-4ba5-bf61-1197626af667\") " pod="openshift-insights/insights-runtime-extractor-5qwfc"
Jul 31 23:46:54.266114 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.266104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f1719f9-a204-4ba5-bf61-1197626af667-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qwfc\" (UID: \"7f1719f9-a204-4ba5-bf61-1197626af667\") " pod="openshift-insights/insights-runtime-extractor-5qwfc"
Jul 31 23:46:54.266392 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.266129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl97c\" (UniqueName: \"kubernetes.io/projected/7f1719f9-a204-4ba5-bf61-1197626af667-kube-api-access-tl97c\") pod \"insights-runtime-extractor-5qwfc\" (UID: \"7f1719f9-a204-4ba5-bf61-1197626af667\") " pod="openshift-insights/insights-runtime-extractor-5qwfc"
Jul 31 23:46:54.266392 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.266154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbvlx\" (UniqueName: \"kubernetes.io/projected/1de6b043-026d-4c14-a302-25c6357dc375-kube-api-access-nbvlx\") pod \"dns-default-5nwm2\" (UID: \"1de6b043-026d-4c14-a302-25c6357dc375\") " pod="openshift-dns/dns-default-5nwm2"
Jul 31 23:46:54.266392 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.266299    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1de6b043-026d-4c14-a302-25c6357dc375-tmp-dir\") pod \"dns-default-5nwm2\" (UID: \"1de6b043-026d-4c14-a302-25c6357dc375\") " pod="openshift-dns/dns-default-5nwm2"
Jul 31 23:46:54.266520 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.266462    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1de6b043-026d-4c14-a302-25c6357dc375-config-volume\") pod \"dns-default-5nwm2\" (UID: \"1de6b043-026d-4c14-a302-25c6357dc375\") " pod="openshift-dns/dns-default-5nwm2"
Jul 31 23:46:54.270102 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.270080    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1de6b043-026d-4c14-a302-25c6357dc375-metrics-tls\") pod \"dns-default-5nwm2\" (UID: \"1de6b043-026d-4c14-a302-25c6357dc375\") " pod="openshift-dns/dns-default-5nwm2"
Jul 31 23:46:54.270208 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.270153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93a82855-64d9-4eb2-bdc5-fb4cfef63f88-cert\") pod \"ingress-canary-gvwvq\" (UID: \"93a82855-64d9-4eb2-bdc5-fb4cfef63f88\") " pod="openshift-ingress-canary/ingress-canary-gvwvq"
Jul 31 23:46:54.274889 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.274861    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbvlx\" (UniqueName: \"kubernetes.io/projected/1de6b043-026d-4c14-a302-25c6357dc375-kube-api-access-nbvlx\") pod \"dns-default-5nwm2\" (UID: \"1de6b043-026d-4c14-a302-25c6357dc375\") " pod="openshift-dns/dns-default-5nwm2"
Jul 31 23:46:54.275387 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.275359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsbjp\" (UniqueName: \"kubernetes.io/projected/93a82855-64d9-4eb2-bdc5-fb4cfef63f88-kube-api-access-zsbjp\") pod \"ingress-canary-gvwvq\" (UID: \"93a82855-64d9-4eb2-bdc5-fb4cfef63f88\") " pod="openshift-ingress-canary/ingress-canary-gvwvq"
Jul 31 23:46:54.367372 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.367338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f1719f9-a204-4ba5-bf61-1197626af667-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qwfc\" (UID: \"7f1719f9-a204-4ba5-bf61-1197626af667\") " pod="openshift-insights/insights-runtime-extractor-5qwfc"
Jul 31 23:46:54.367372 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.367386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f1719f9-a204-4ba5-bf61-1197626af667-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qwfc\" (UID: \"7f1719f9-a204-4ba5-bf61-1197626af667\") " pod="openshift-insights/insights-runtime-extractor-5qwfc"
Jul 31 23:46:54.367625 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.367403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl97c\" (UniqueName: \"kubernetes.io/projected/7f1719f9-a204-4ba5-bf61-1197626af667-kube-api-access-tl97c\") pod \"insights-runtime-extractor-5qwfc\" (UID: \"7f1719f9-a204-4ba5-bf61-1197626af667\") " pod="openshift-insights/insights-runtime-extractor-5qwfc"
Jul 31 23:46:54.367625 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.367427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f1719f9-a204-4ba5-bf61-1197626af667-crio-socket\") pod \"insights-runtime-extractor-5qwfc\" (UID: \"7f1719f9-a204-4ba5-bf61-1197626af667\") " pod="openshift-insights/insights-runtime-extractor-5qwfc"
Jul 31 23:46:54.367625 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.367457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f1719f9-a204-4ba5-bf61-1197626af667-data-volume\") pod \"insights-runtime-extractor-5qwfc\" (UID: \"7f1719f9-a204-4ba5-bf61-1197626af667\") " pod="openshift-insights/insights-runtime-extractor-5qwfc"
Jul 31 23:46:54.367625 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.367526    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f1719f9-a204-4ba5-bf61-1197626af667-crio-socket\") pod \"insights-runtime-extractor-5qwfc\" (UID: \"7f1719f9-a204-4ba5-bf61-1197626af667\") " pod="openshift-insights/insights-runtime-extractor-5qwfc"
Jul 31 23:46:54.367852 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.367837    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f1719f9-a204-4ba5-bf61-1197626af667-data-volume\") pod \"insights-runtime-extractor-5qwfc\" (UID: \"7f1719f9-a204-4ba5-bf61-1197626af667\") " pod="openshift-insights/insights-runtime-extractor-5qwfc"
Jul 31 23:46:54.368505 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.368489    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f1719f9-a204-4ba5-bf61-1197626af667-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qwfc\" (UID: \"7f1719f9-a204-4ba5-bf61-1197626af667\") " pod="openshift-insights/insights-runtime-extractor-5qwfc"
Jul 31 23:46:54.369871 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.369852    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f1719f9-a204-4ba5-bf61-1197626af667-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qwfc\" (UID: \"7f1719f9-a204-4ba5-bf61-1197626af667\") " pod="openshift-insights/insights-runtime-extractor-5qwfc"
Jul 31 23:46:54.377292 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.377271    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl97c\" (UniqueName: \"kubernetes.io/projected/7f1719f9-a204-4ba5-bf61-1197626af667-kube-api-access-tl97c\") pod \"insights-runtime-extractor-5qwfc\" (UID: \"7f1719f9-a204-4ba5-bf61-1197626af667\") " pod="openshift-insights/insights-runtime-extractor-5qwfc"
Jul 31 23:46:54.385232 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.385207    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5nwm2"
Jul 31 23:46:54.402812 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.402789    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gvwvq"
Jul 31 23:46:54.466740 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.466682    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5qwfc"
Jul 31 23:46:54.543138 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.542330    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5nwm2"]
Jul 31 23:46:54.552269 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:46:54.552220    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de6b043_026d_4c14_a302_25c6357dc375.slice/crio-b53c272a2b8b40a8ece298027eaee9d203732092f1bd2d7e694a1caed06ce2fd WatchSource:0}: Error finding container b53c272a2b8b40a8ece298027eaee9d203732092f1bd2d7e694a1caed06ce2fd: Status 404 returned error can't find the container with id b53c272a2b8b40a8ece298027eaee9d203732092f1bd2d7e694a1caed06ce2fd
Jul 31 23:46:54.574915 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.574891    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gvwvq"]
Jul 31 23:46:54.580230 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:46:54.580198    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a82855_64d9_4eb2_bdc5_fb4cfef63f88.slice/crio-fb44ac2fdbeece453ce5150d8630dc3f660ad0276a153674d4d95adc730b88b9 WatchSource:0}: Error finding container fb44ac2fdbeece453ce5150d8630dc3f660ad0276a153674d4d95adc730b88b9: Status 404 returned error can't find the container with id fb44ac2fdbeece453ce5150d8630dc3f660ad0276a153674d4d95adc730b88b9
Jul 31 23:46:54.650208 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:54.650182    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5qwfc"]
Jul 31 23:46:54.653679 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:46:54.653646    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1719f9_a204_4ba5_bf61_1197626af667.slice/crio-befe4d740096612e7773ed24bcb494279ae564875b78044b49d1ff120ca30225 WatchSource:0}: Error finding container befe4d740096612e7773ed24bcb494279ae564875b78044b49d1ff120ca30225: Status 404 returned error can't find the container with id befe4d740096612e7773ed24bcb494279ae564875b78044b49d1ff120ca30225
Jul 31 23:46:55.180899 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:55.180853    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:46:55.181084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:55.180863    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:46:55.181084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:55.181017    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:46:55.185110 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:55.185085    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 23:46:55.185468 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:55.185127    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 23:46:55.185747 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:55.185730    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c2rbb\""
Jul 31 23:46:55.186072 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:55.186052    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 23:46:55.186293 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:55.186166    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 23:46:55.186478 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:55.186460    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45dqd\""
Jul 31 23:46:55.401198 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:55.401090    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5nwm2" event={"ID":"1de6b043-026d-4c14-a302-25c6357dc375","Type":"ContainerStarted","Data":"b53c272a2b8b40a8ece298027eaee9d203732092f1bd2d7e694a1caed06ce2fd"}
Jul 31 23:46:55.403514 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:55.403474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5qwfc" event={"ID":"7f1719f9-a204-4ba5-bf61-1197626af667","Type":"ContainerStarted","Data":"f5aace9f81534ac0db13d7c69a3906943f0d5c0bf47ff6a2da3a578ed1d21844"}
Jul 31 23:46:55.403675 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:55.403564    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5qwfc" event={"ID":"7f1719f9-a204-4ba5-bf61-1197626af667","Type":"ContainerStarted","Data":"93b76fddd243c7c0851e1a2210022746673567d731ec784bbaa2ba06e4bf4ff7"}
Jul 31 23:46:55.403805 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:55.403700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5qwfc" event={"ID":"7f1719f9-a204-4ba5-bf61-1197626af667","Type":"ContainerStarted","Data":"befe4d740096612e7773ed24bcb494279ae564875b78044b49d1ff120ca30225"}
Jul 31 23:46:55.405101 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:55.405059    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gvwvq" event={"ID":"93a82855-64d9-4eb2-bdc5-fb4cfef63f88","Type":"ContainerStarted","Data":"fb44ac2fdbeece453ce5150d8630dc3f660ad0276a153674d4d95adc730b88b9"}
Jul 31 23:46:56.967834 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:56.967806    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bcf7bf76f-s6hlw"]
Jul 31 23:46:56.970844 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:56.970814    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:56.975330 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:56.973167    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 31 23:46:56.975330 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:56.973313    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 31 23:46:56.975330 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:56.973572    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 31 23:46:56.975330 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:56.973591    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 31 23:46:56.975330 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:56.973687    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 31 23:46:56.975867 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:56.975838    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 31 23:46:56.975984 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:56.975938    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 31 23:46:56.976052 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:56.976007    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xsm47\""
Jul 31 23:46:56.978245 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:56.978219    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bcf7bf76f-s6hlw"]
Jul 31 23:46:56.989672 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:56.989644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4f15005-6696-47b0-9a66-15e23fbe88e2-console-oauth-config\") pod \"console-bcf7bf76f-s6hlw\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") " pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:56.989794 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:56.989737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4f15005-6696-47b0-9a66-15e23fbe88e2-console-config\") pod \"console-bcf7bf76f-s6hlw\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") " pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:56.989794 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:56.989768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f15005-6696-47b0-9a66-15e23fbe88e2-service-ca\") pod \"console-bcf7bf76f-s6hlw\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") " pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:56.989918 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:56.989814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtl27\" (UniqueName: \"kubernetes.io/projected/b4f15005-6696-47b0-9a66-15e23fbe88e2-kube-api-access-mtl27\") pod \"console-bcf7bf76f-s6hlw\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") " pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:56.989918 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:56.989875    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4f15005-6696-47b0-9a66-15e23fbe88e2-console-serving-cert\") pod \"console-bcf7bf76f-s6hlw\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") " pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:56.990021 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:56.989944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4f15005-6696-47b0-9a66-15e23fbe88e2-oauth-serving-cert\") pod \"console-bcf7bf76f-s6hlw\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") " pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:57.090453 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.090413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4f15005-6696-47b0-9a66-15e23fbe88e2-oauth-serving-cert\") pod \"console-bcf7bf76f-s6hlw\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") " pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:57.090641 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.090470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4f15005-6696-47b0-9a66-15e23fbe88e2-console-oauth-config\") pod \"console-bcf7bf76f-s6hlw\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") " pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:57.090641 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.090514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4f15005-6696-47b0-9a66-15e23fbe88e2-console-config\") pod \"console-bcf7bf76f-s6hlw\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") " pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:57.090641 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.090532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f15005-6696-47b0-9a66-15e23fbe88e2-service-ca\") pod \"console-bcf7bf76f-s6hlw\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") " pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:57.090641 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.090563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtl27\" (UniqueName: \"kubernetes.io/projected/b4f15005-6696-47b0-9a66-15e23fbe88e2-kube-api-access-mtl27\") pod \"console-bcf7bf76f-s6hlw\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") " pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:57.090641 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.090597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4f15005-6696-47b0-9a66-15e23fbe88e2-console-serving-cert\") pod \"console-bcf7bf76f-s6hlw\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") " pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:57.091277 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.091227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4f15005-6696-47b0-9a66-15e23fbe88e2-oauth-serving-cert\") pod \"console-bcf7bf76f-s6hlw\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") " pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:57.091630 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.091607    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f15005-6696-47b0-9a66-15e23fbe88e2-service-ca\") pod \"console-bcf7bf76f-s6hlw\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") " pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:57.092188 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.092144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4f15005-6696-47b0-9a66-15e23fbe88e2-console-config\") pod \"console-bcf7bf76f-s6hlw\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") " pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:57.093544 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.093524    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4f15005-6696-47b0-9a66-15e23fbe88e2-console-serving-cert\") pod \"console-bcf7bf76f-s6hlw\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") " pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:57.093544 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.093536    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4f15005-6696-47b0-9a66-15e23fbe88e2-console-oauth-config\") pod \"console-bcf7bf76f-s6hlw\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") " pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:57.102543 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.102518    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtl27\" (UniqueName: \"kubernetes.io/projected/b4f15005-6696-47b0-9a66-15e23fbe88e2-kube-api-access-mtl27\") pod \"console-bcf7bf76f-s6hlw\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") " pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:57.283943 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.283863    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:46:57.445557 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.445165    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-dlctp"]
Jul 31 23:46:57.449981 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.449569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp"
Jul 31 23:46:57.453026 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.452936    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 31 23:46:57.454915 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.453276    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 31 23:46:57.454915 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.453538    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 31 23:46:57.454915 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.453780    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 31 23:46:57.454915 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.453978    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-x4xvw\""
Jul 31 23:46:57.454915 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.454689    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 31 23:46:57.462174 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.462144    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-dlctp"]
Jul 31 23:46:57.483682 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.483602    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bcf7bf76f-s6hlw"]
Jul 31 23:46:57.492936 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.492908    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a162687-ed57-4042-b0f8-8a41623fa94c-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-dlctp\" (UID: \"7a162687-ed57-4042-b0f8-8a41623fa94c\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp"
Jul 31 23:46:57.493103 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.492973    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a162687-ed57-4042-b0f8-8a41623fa94c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-dlctp\" (UID: \"7a162687-ed57-4042-b0f8-8a41623fa94c\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp"
Jul 31 23:46:57.493103 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.493005    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l54f\" (UniqueName: \"kubernetes.io/projected/7a162687-ed57-4042-b0f8-8a41623fa94c-kube-api-access-2l54f\") pod \"prometheus-operator-758d7cb874-dlctp\" (UID: \"7a162687-ed57-4042-b0f8-8a41623fa94c\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp"
Jul 31 23:46:57.493103 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.493049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a162687-ed57-4042-b0f8-8a41623fa94c-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-dlctp\" (UID: \"7a162687-ed57-4042-b0f8-8a41623fa94c\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp"
Jul 31 23:46:57.493641 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:46:57.493601    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f15005_6696_47b0_9a66_15e23fbe88e2.slice/crio-dc2c5bd5e25dbdf1877b7517e175cd6fadfc39e6b769329152c097b7b01f964a WatchSource:0}: Error finding container dc2c5bd5e25dbdf1877b7517e175cd6fadfc39e6b769329152c097b7b01f964a: Status 404 returned error can't find the container with id dc2c5bd5e25dbdf1877b7517e175cd6fadfc39e6b769329152c097b7b01f964a
Jul 31 23:46:57.594147 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.594047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a162687-ed57-4042-b0f8-8a41623fa94c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-dlctp\" (UID: \"7a162687-ed57-4042-b0f8-8a41623fa94c\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp"
Jul 31 23:46:57.594147 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.594100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l54f\" (UniqueName: \"kubernetes.io/projected/7a162687-ed57-4042-b0f8-8a41623fa94c-kube-api-access-2l54f\") pod \"prometheus-operator-758d7cb874-dlctp\" (UID: \"7a162687-ed57-4042-b0f8-8a41623fa94c\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp"
Jul 31 23:46:57.594147 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.594138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a162687-ed57-4042-b0f8-8a41623fa94c-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-dlctp\" (UID: \"7a162687-ed57-4042-b0f8-8a41623fa94c\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp"
Jul 31 23:46:57.594503 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.594216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a162687-ed57-4042-b0f8-8a41623fa94c-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-dlctp\" (UID: \"7a162687-ed57-4042-b0f8-8a41623fa94c\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp"
Jul 31 23:46:57.594503 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:57.594308    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 31 23:46:57.594503 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:46:57.594388    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a162687-ed57-4042-b0f8-8a41623fa94c-prometheus-operator-tls podName:7a162687-ed57-4042-b0f8-8a41623fa94c nodeName:}" failed. No retries permitted until 2026-07-31 23:46:58.094366392 +0000 UTC m=+57.454257412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/7a162687-ed57-4042-b0f8-8a41623fa94c-prometheus-operator-tls") pod "prometheus-operator-758d7cb874-dlctp" (UID: "7a162687-ed57-4042-b0f8-8a41623fa94c") : secret "prometheus-operator-tls" not found
Jul 31 23:46:57.594928 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.594909    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a162687-ed57-4042-b0f8-8a41623fa94c-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-dlctp\" (UID: \"7a162687-ed57-4042-b0f8-8a41623fa94c\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp"
Jul 31 23:46:57.596617 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.596594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a162687-ed57-4042-b0f8-8a41623fa94c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-dlctp\" (UID: \"7a162687-ed57-4042-b0f8-8a41623fa94c\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp"
Jul 31 23:46:57.604284 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:57.604266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l54f\" (UniqueName: \"kubernetes.io/projected/7a162687-ed57-4042-b0f8-8a41623fa94c-kube-api-access-2l54f\") pod \"prometheus-operator-758d7cb874-dlctp\" (UID: \"7a162687-ed57-4042-b0f8-8a41623fa94c\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp"
Jul 31 23:46:58.099470 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:58.099434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a162687-ed57-4042-b0f8-8a41623fa94c-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-dlctp\" (UID: \"7a162687-ed57-4042-b0f8-8a41623fa94c\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp"
Jul 31 23:46:58.102750 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:58.102600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a162687-ed57-4042-b0f8-8a41623fa94c-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-dlctp\" (UID: \"7a162687-ed57-4042-b0f8-8a41623fa94c\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp"
Jul 31 23:46:58.364999 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:58.364920    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp"
Jul 31 23:46:58.419145 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:58.419017    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5qwfc" event={"ID":"7f1719f9-a204-4ba5-bf61-1197626af667","Type":"ContainerStarted","Data":"65bb337d2f0a133ca21200c0f423b55f1f5333d6be981b373346a25f6b475393"}
Jul 31 23:46:58.422332 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:58.422282    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bcf7bf76f-s6hlw" event={"ID":"b4f15005-6696-47b0-9a66-15e23fbe88e2","Type":"ContainerStarted","Data":"dc2c5bd5e25dbdf1877b7517e175cd6fadfc39e6b769329152c097b7b01f964a"}
Jul 31 23:46:58.425925 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:58.425271    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5nwm2" event={"ID":"1de6b043-026d-4c14-a302-25c6357dc375","Type":"ContainerStarted","Data":"253348245b347bc538a6ae6a3633d0c6d9f25695e8b426091f412aff31b6c6e9"}
Jul 31 23:46:58.425925 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:58.425330    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5nwm2" event={"ID":"1de6b043-026d-4c14-a302-25c6357dc375","Type":"ContainerStarted","Data":"707c9401f430290d1617220dd78ce93a91fcef715b9226ec9fda3d4dcdb77530"}
Jul 31 23:46:58.425925 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:58.425856    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5nwm2"
Jul 31 23:46:58.428492 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:58.428469    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gvwvq" event={"ID":"93a82855-64d9-4eb2-bdc5-fb4cfef63f88","Type":"ContainerStarted","Data":"0ecc9718e922cb5755038d29457d6858f6ff2bbd5f85dbb94f065e167fe87a2e"}
Jul 31 23:46:58.440758 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:58.440687    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5qwfc" podStartSLOduration=1.859456215 podStartE2EDuration="4.440671589s" totalImagesPullingTime="2.581215374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:58.439525723 +0000 UTC m=+57.799416801" watchObservedRunningTime="2026-07-31 23:46:58.440671589 +0000 UTC m=+57.800562630"
Jul 31 23:46:58.457461 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:58.457410    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gvwvq" podStartSLOduration=1.717350266 podStartE2EDuration="4.457391331s" totalImagesPullingTime="2.740041065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:58.456116929 +0000 UTC m=+57.816007971" watchObservedRunningTime="2026-07-31 23:46:58.457391331 +0000 UTC m=+57.817282350"
Jul 31 23:46:58.475943 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:58.475886    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5nwm2" podStartSLOduration=1.708539578 podStartE2EDuration="4.475865669s" totalImagesPullingTime="2.767326091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:58.473816569 +0000 UTC m=+57.833707638" watchObservedRunningTime="2026-07-31 23:46:58.475865669 +0000 UTC m=+57.835756711"
Jul 31 23:46:58.512610 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:58.512578    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-dlctp"]
Jul 31 23:46:58.520739 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:46:58.517956    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a162687_ed57_4042_b0f8_8a41623fa94c.slice/crio-4e0549012aba92885664c7dc67228ee2928d7f23f3dcfcaf3516038891af7f88 WatchSource:0}: Error finding container 4e0549012aba92885664c7dc67228ee2928d7f23f3dcfcaf3516038891af7f88: Status 404 returned error can't find the container with id 4e0549012aba92885664c7dc67228ee2928d7f23f3dcfcaf3516038891af7f88
Jul 31 23:46:59.434113 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:46:59.434078    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp" event={"ID":"7a162687-ed57-4042-b0f8-8a41623fa94c","Type":"ContainerStarted","Data":"4e0549012aba92885664c7dc67228ee2928d7f23f3dcfcaf3516038891af7f88"}
Jul 31 23:47:01.441495 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:01.441455    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bcf7bf76f-s6hlw" event={"ID":"b4f15005-6696-47b0-9a66-15e23fbe88e2","Type":"ContainerStarted","Data":"1496d35c917fa5a3af7551dce3ce79204dc2d4730ffcb5d5626bb5452c6e90cf"}
Jul 31 23:47:01.443394 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:01.443358    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp" event={"ID":"7a162687-ed57-4042-b0f8-8a41623fa94c","Type":"ContainerStarted","Data":"08cc3b3a824133b91ecadc11ea43e00355631fff344b792fcdb8691e6ccacf40"}
Jul 31 23:47:01.443526 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:01.443408    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp" event={"ID":"7a162687-ed57-4042-b0f8-8a41623fa94c","Type":"ContainerStarted","Data":"4bff13fb56eb4ecbdbc46b5809d8988452511607e03e6107e95961e6d5ec07a0"}
Jul 31 23:47:01.456591 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:01.456537    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bcf7bf76f-s6hlw" podStartSLOduration=2.371012445 podStartE2EDuration="5.456521959s" totalImagesPullingTime="3.085509514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:01.455860168 +0000 UTC m=+60.815751208" watchObservedRunningTime="2026-07-31 23:47:01.456521959 +0000 UTC m=+60.816413000"
Jul 31 23:47:03.355387 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.355361    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xdzrq"
Jul 31 23:47:03.383419 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.383083    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-dlctp" podStartSLOduration=4.321748105 podStartE2EDuration="6.383061956s" totalImagesPullingTime="2.061313851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:01.471154864 +0000 UTC m=+60.831045905" watchObservedRunningTime="2026-07-31 23:47:03.383061956 +0000 UTC m=+62.742952996"
Jul 31 23:47:03.785699 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.785670    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8"]
Jul 31 23:47:03.790025 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.790007    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8"
Jul 31 23:47:03.792266 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.792244    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fj5hm\""
Jul 31 23:47:03.792423 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.792405    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 31 23:47:03.792617 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.792603    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 31 23:47:03.799986 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.799964    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8"]
Jul 31 23:47:03.805024 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.805000    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"]
Jul 31 23:47:03.808659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.808636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:03.811011 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.810989    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zwz8j\""
Jul 31 23:47:03.811085 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.811045    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 31 23:47:03.811156 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.811137    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 31 23:47:03.811206 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.811170    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 31 23:47:03.824161 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.824135    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-75gvm"]
Jul 31 23:47:03.827541 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.827524    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"]
Jul 31 23:47:03.827652 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.827639    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:03.829419 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.829393    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 31 23:47:03.829419 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.829410    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 31 23:47:03.829565 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.829458    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 31 23:47:03.829565 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.829412    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f4bpr\""
Jul 31 23:47:03.840480 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.840460    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tvz6\" (UniqueName: \"kubernetes.io/projected/46bbcd5c-1c67-4966-8290-90106568f72b-kube-api-access-4tvz6\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:03.840565 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.840499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6cbde346-ebc0-415d-85e7-c61319db6262-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-lkdp8\" (UID: \"6cbde346-ebc0-415d-85e7-c61319db6262\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8"
Jul 31 23:47:03.840565 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.840516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg4xc\" (UniqueName: \"kubernetes.io/projected/6cbde346-ebc0-415d-85e7-c61319db6262-kube-api-access-pg4xc\") pod \"openshift-state-metrics-6bb4d7b4dd-lkdp8\" (UID: \"6cbde346-ebc0-415d-85e7-c61319db6262\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8"
Jul 31 23:47:03.840630 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.840590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/46bbcd5c-1c67-4966-8290-90106568f72b-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:03.840661 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.840636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/46bbcd5c-1c67-4966-8290-90106568f72b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:03.840698 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.840661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46bbcd5c-1c67-4966-8290-90106568f72b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:03.840698 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.840679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cbde346-ebc0-415d-85e7-c61319db6262-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-lkdp8\" (UID: \"6cbde346-ebc0-415d-85e7-c61319db6262\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8"
Jul 31 23:47:03.840781 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.840703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46bbcd5c-1c67-4966-8290-90106568f72b-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:03.840781 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.840749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6cbde346-ebc0-415d-85e7-c61319db6262-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-lkdp8\" (UID: \"6cbde346-ebc0-415d-85e7-c61319db6262\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8"
Jul 31 23:47:03.840843 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.840796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46bbcd5c-1c67-4966-8290-90106568f72b-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:03.941527 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.941486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6cbde346-ebc0-415d-85e7-c61319db6262-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-lkdp8\" (UID: \"6cbde346-ebc0-415d-85e7-c61319db6262\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8"
Jul 31 23:47:03.941527 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.941524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg4xc\" (UniqueName: \"kubernetes.io/projected/6cbde346-ebc0-415d-85e7-c61319db6262-kube-api-access-pg4xc\") pod \"openshift-state-metrics-6bb4d7b4dd-lkdp8\" (UID: \"6cbde346-ebc0-415d-85e7-c61319db6262\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8"
Jul 31 23:47:03.941780 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.941568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/46bbcd5c-1c67-4966-8290-90106568f72b-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:03.941780 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.941603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/46bbcd5c-1c67-4966-8290-90106568f72b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:03.941780 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.941634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2e5bad06-e384-45d4-8863-7d2563e0d648-node-exporter-textfile\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:03.941780 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.941661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2e5bad06-e384-45d4-8863-7d2563e0d648-node-exporter-accelerators-collector-config\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:03.941780 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.941740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46bbcd5c-1c67-4966-8290-90106568f72b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:03.941780 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.941766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cbde346-ebc0-415d-85e7-c61319db6262-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-lkdp8\" (UID: \"6cbde346-ebc0-415d-85e7-c61319db6262\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8"
Jul 31 23:47:03.942075 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.941791    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46bbcd5c-1c67-4966-8290-90106568f72b-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:03.942075 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.941816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6cbde346-ebc0-415d-85e7-c61319db6262-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-lkdp8\" (UID: \"6cbde346-ebc0-415d-85e7-c61319db6262\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8"
Jul 31 23:47:03.942075 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.942054    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/46bbcd5c-1c67-4966-8290-90106568f72b-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:03.942285 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.942263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e5bad06-e384-45d4-8863-7d2563e0d648-sys\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:03.942479 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.942463    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e5bad06-e384-45d4-8863-7d2563e0d648-metrics-client-ca\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:03.942605 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.942515    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46bbcd5c-1c67-4966-8290-90106568f72b-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:03.942776 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.942597    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e5bad06-e384-45d4-8863-7d2563e0d648-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:03.942869 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.942791    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cbde346-ebc0-415d-85e7-c61319db6262-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-lkdp8\" (UID: \"6cbde346-ebc0-415d-85e7-c61319db6262\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8"
Jul 31 23:47:03.942869 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.942614    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/46bbcd5c-1c67-4966-8290-90106568f72b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:03.942869 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.942810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46bbcd5c-1c67-4966-8290-90106568f72b-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:03.942869 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.942864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2e5bad06-e384-45d4-8863-7d2563e0d648-root\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:03.943036 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:47:03.942896    2578 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 31 23:47:03.943036 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.942903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22g2k\" (UniqueName: \"kubernetes.io/projected/2e5bad06-e384-45d4-8863-7d2563e0d648-kube-api-access-22g2k\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:03.943036 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:47:03.942958    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46bbcd5c-1c67-4966-8290-90106568f72b-kube-state-metrics-tls podName:46bbcd5c-1c67-4966-8290-90106568f72b nodeName:}" failed. No retries permitted until 2026-07-31 23:47:04.442937156 +0000 UTC m=+63.802828191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/46bbcd5c-1c67-4966-8290-90106568f72b-kube-state-metrics-tls") pod "kube-state-metrics-6bd9664c4-r4f5l" (UID: "46bbcd5c-1c67-4966-8290-90106568f72b") : secret "kube-state-metrics-tls" not found
Jul 31 23:47:03.943036 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.943001    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e5bad06-e384-45d4-8863-7d2563e0d648-node-exporter-tls\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:03.943036 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.943028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2e5bad06-e384-45d4-8863-7d2563e0d648-node-exporter-wtmp\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:03.943218 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.943059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tvz6\" (UniqueName: \"kubernetes.io/projected/46bbcd5c-1c67-4966-8290-90106568f72b-kube-api-access-4tvz6\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:03.944255 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.944231    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6cbde346-ebc0-415d-85e7-c61319db6262-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-lkdp8\" (UID: \"6cbde346-ebc0-415d-85e7-c61319db6262\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8"
Jul 31 23:47:03.944339 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.944279    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6cbde346-ebc0-415d-85e7-c61319db6262-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-lkdp8\" (UID: \"6cbde346-ebc0-415d-85e7-c61319db6262\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8"
Jul 31 23:47:03.944401 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.944376    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46bbcd5c-1c67-4966-8290-90106568f72b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:03.951334 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.951314    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg4xc\" (UniqueName: \"kubernetes.io/projected/6cbde346-ebc0-415d-85e7-c61319db6262-kube-api-access-pg4xc\") pod \"openshift-state-metrics-6bb4d7b4dd-lkdp8\" (UID: \"6cbde346-ebc0-415d-85e7-c61319db6262\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8"
Jul 31 23:47:03.953558 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:03.953532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tvz6\" (UniqueName: \"kubernetes.io/projected/46bbcd5c-1c67-4966-8290-90106568f72b-kube-api-access-4tvz6\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:04.043997 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.043900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2e5bad06-e384-45d4-8863-7d2563e0d648-node-exporter-textfile\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.043997 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.043947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2e5bad06-e384-45d4-8863-7d2563e0d648-node-exporter-accelerators-collector-config\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.043997 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.043968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e5bad06-e384-45d4-8863-7d2563e0d648-sys\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.043997 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.043986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e5bad06-e384-45d4-8863-7d2563e0d648-metrics-client-ca\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.043997 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.044002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e5bad06-e384-45d4-8863-7d2563e0d648-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.044352 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.044029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2e5bad06-e384-45d4-8863-7d2563e0d648-root\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.044352 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.044046    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e5bad06-e384-45d4-8863-7d2563e0d648-sys\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.044352 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.044053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22g2k\" (UniqueName: \"kubernetes.io/projected/2e5bad06-e384-45d4-8863-7d2563e0d648-kube-api-access-22g2k\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.044352 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.044109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2e5bad06-e384-45d4-8863-7d2563e0d648-root\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.044352 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.044153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e5bad06-e384-45d4-8863-7d2563e0d648-node-exporter-tls\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.044352 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.044180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2e5bad06-e384-45d4-8863-7d2563e0d648-node-exporter-wtmp\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.044352 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:47:04.044267    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 31 23:47:04.044352 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:47:04.044337    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e5bad06-e384-45d4-8863-7d2563e0d648-node-exporter-tls podName:2e5bad06-e384-45d4-8863-7d2563e0d648 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:04.544319336 +0000 UTC m=+63.904210355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2e5bad06-e384-45d4-8863-7d2563e0d648-node-exporter-tls") pod "node-exporter-75gvm" (UID: "2e5bad06-e384-45d4-8863-7d2563e0d648") : secret "node-exporter-tls" not found
Jul 31 23:47:04.044352 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.044347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2e5bad06-e384-45d4-8863-7d2563e0d648-node-exporter-textfile\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.044352 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.044355    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2e5bad06-e384-45d4-8863-7d2563e0d648-node-exporter-wtmp\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.044839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.044572    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2e5bad06-e384-45d4-8863-7d2563e0d648-node-exporter-accelerators-collector-config\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.044839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.044666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e5bad06-e384-45d4-8863-7d2563e0d648-metrics-client-ca\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.046608 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.046590    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e5bad06-e384-45d4-8863-7d2563e0d648-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.052845 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.052823    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22g2k\" (UniqueName: \"kubernetes.io/projected/2e5bad06-e384-45d4-8863-7d2563e0d648-kube-api-access-22g2k\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.098926 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.098888    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8"
Jul 31 23:47:04.241696 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.241664    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8"]
Jul 31 23:47:04.244784 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:47:04.244757    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cbde346_ebc0_415d_85e7_c61319db6262.slice/crio-eeb0da477f33f9dde3416ec8e40c75e84d34c70232fde776052612380dc3c50d WatchSource:0}: Error finding container eeb0da477f33f9dde3416ec8e40c75e84d34c70232fde776052612380dc3c50d: Status 404 returned error can't find the container with id eeb0da477f33f9dde3416ec8e40c75e84d34c70232fde776052612380dc3c50d
Jul 31 23:47:04.446902 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.446853    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46bbcd5c-1c67-4966-8290-90106568f72b-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:04.449460 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.449437    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46bbcd5c-1c67-4966-8290-90106568f72b-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-r4f5l\" (UID: \"46bbcd5c-1c67-4966-8290-90106568f72b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:04.454014 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.453988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8" event={"ID":"6cbde346-ebc0-415d-85e7-c61319db6262","Type":"ContainerStarted","Data":"56c1f2005c428dd6f355655ac5fe7cdcdbf52ac92009dda44825c4cbfc1dab42"}
Jul 31 23:47:04.454116 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.454040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8" event={"ID":"6cbde346-ebc0-415d-85e7-c61319db6262","Type":"ContainerStarted","Data":"f3f32427061538965d3666a543939d63e6ae15c0392b5644ea2bbbf62f85595d"}
Jul 31 23:47:04.454116 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.454060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8" event={"ID":"6cbde346-ebc0-415d-85e7-c61319db6262","Type":"ContainerStarted","Data":"eeb0da477f33f9dde3416ec8e40c75e84d34c70232fde776052612380dc3c50d"}
Jul 31 23:47:04.548286 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.548242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e5bad06-e384-45d4-8863-7d2563e0d648-node-exporter-tls\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.550783 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.550694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e5bad06-e384-45d4-8863-7d2563e0d648-node-exporter-tls\") pod \"node-exporter-75gvm\" (UID: \"2e5bad06-e384-45d4-8863-7d2563e0d648\") " pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.717084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.717050    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"
Jul 31 23:47:04.735562 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.735534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-75gvm"
Jul 31 23:47:04.747263 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:47:04.747225    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5bad06_e384_45d4_8863_7d2563e0d648.slice/crio-e038aba874862ff5c8628e6eb66ecbc1b3c09a1450378d4bd0155212815605f3 WatchSource:0}: Error finding container e038aba874862ff5c8628e6eb66ecbc1b3c09a1450378d4bd0155212815605f3: Status 404 returned error can't find the container with id e038aba874862ff5c8628e6eb66ecbc1b3c09a1450378d4bd0155212815605f3
Jul 31 23:47:04.853262 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:04.853236    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l"]
Jul 31 23:47:04.856830 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:47:04.856787    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bbcd5c_1c67_4966_8290_90106568f72b.slice/crio-f5c21f8692f360eff4110379c5be0f3ec368a82690360be4e2f0a297da8c5e06 WatchSource:0}: Error finding container f5c21f8692f360eff4110379c5be0f3ec368a82690360be4e2f0a297da8c5e06: Status 404 returned error can't find the container with id f5c21f8692f360eff4110379c5be0f3ec368a82690360be4e2f0a297da8c5e06
Jul 31 23:47:05.466169 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:05.466099    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l" event={"ID":"46bbcd5c-1c67-4966-8290-90106568f72b","Type":"ContainerStarted","Data":"f5c21f8692f360eff4110379c5be0f3ec368a82690360be4e2f0a297da8c5e06"}
Jul 31 23:47:05.469464 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:05.469427    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8" event={"ID":"6cbde346-ebc0-415d-85e7-c61319db6262","Type":"ContainerStarted","Data":"0e54b9bcc6987247fc2d795bfd6912971bdd33b83716327bdec6f1d64c5fd450"}
Jul 31 23:47:05.471538 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:05.471512    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75gvm" event={"ID":"2e5bad06-e384-45d4-8863-7d2563e0d648","Type":"ContainerStarted","Data":"e038aba874862ff5c8628e6eb66ecbc1b3c09a1450378d4bd0155212815605f3"}
Jul 31 23:47:05.489928 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:05.489874    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lkdp8" podStartSLOduration=1.464921072 podStartE2EDuration="2.489857448s" totalImagesPullingTime="1.024936376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:05.488660642 +0000 UTC m=+64.848551739" watchObservedRunningTime="2026-07-31 23:47:05.489857448 +0000 UTC m=+64.849748489"
Jul 31 23:47:06.758189 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.758159    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-579d944d7c-gk9x9"]
Jul 31 23:47:06.772108 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.772082    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.772263 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.772134    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-579d944d7c-gk9x9"]
Jul 31 23:47:06.779826 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.779703    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 31 23:47:06.868324 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.868294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-trusted-ca-bundle\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.868324 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.868325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/311b3b3f-026b-495e-8b94-6943662fe25e-console-serving-cert\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.868536 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.868354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs\") pod \"network-metrics-daemon-js2f5\" (UID: \"ec95cdf0-c411-409a-a0e5-274712ef6c00\") " pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:47:06.868536 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.868395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pqmh\" (UniqueName: \"kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh\") pod \"network-check-target-mjkm4\" (UID: \"824b4cc2-7b16-43c6-9cc0-46759ffdf9b5\") " pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:47:06.868536 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.868415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-oauth-serving-cert\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.868536 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.868438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/311b3b3f-026b-495e-8b94-6943662fe25e-console-oauth-config\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.868536 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.868460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret\") pod \"global-pull-secret-syncer-zzsl6\" (UID: \"ab5415d7-3b45-4861-bfaf-be67c4d16871\") " pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:47:06.868536 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.868492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-console-config\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.868801 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.868605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-service-ca\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.868801 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.868638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsbcc\" (UniqueName: \"kubernetes.io/projected/311b3b3f-026b-495e-8b94-6943662fe25e-kube-api-access-jsbcc\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.870314 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.870294    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 23:47:06.870398 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.870339    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 23:47:06.870398 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.870390    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 23:47:06.881470 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.881445    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 23:47:06.882000 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.881977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec95cdf0-c411-409a-a0e5-274712ef6c00-metrics-certs\") pod \"network-metrics-daemon-js2f5\" (UID: \"ec95cdf0-c411-409a-a0e5-274712ef6c00\") " pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:47:06.882155 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.882142    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab5415d7-3b45-4861-bfaf-be67c4d16871-original-pull-secret\") pod \"global-pull-secret-syncer-zzsl6\" (UID: \"ab5415d7-3b45-4861-bfaf-be67c4d16871\") " pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:47:06.892504 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.892478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pqmh\" (UniqueName: \"kubernetes.io/projected/824b4cc2-7b16-43c6-9cc0-46759ffdf9b5-kube-api-access-9pqmh\") pod \"network-check-target-mjkm4\" (UID: \"824b4cc2-7b16-43c6-9cc0-46759ffdf9b5\") " pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:47:06.896669 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.896642    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c2rbb\""
Jul 31 23:47:06.903727 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.903684    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzsl6"
Jul 31 23:47:06.906124 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.906109    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-js2f5"
Jul 31 23:47:06.914156 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.914138    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45dqd\""
Jul 31 23:47:06.922600 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.922564    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:47:06.968963 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.968938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-service-ca\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.968963 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.968966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsbcc\" (UniqueName: \"kubernetes.io/projected/311b3b3f-026b-495e-8b94-6943662fe25e-kube-api-access-jsbcc\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.969219 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.968992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-trusted-ca-bundle\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.969219 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.969018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/311b3b3f-026b-495e-8b94-6943662fe25e-console-serving-cert\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.969219 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.969055    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-oauth-serving-cert\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.969219 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.969083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/311b3b3f-026b-495e-8b94-6943662fe25e-console-oauth-config\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.969219 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.969120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-console-config\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.969936 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.969906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-service-ca\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.970141 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.970115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-trusted-ca-bundle\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.970287 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.970219    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-oauth-serving-cert\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.970432 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.970414    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-console-config\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.972009 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.971987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/311b3b3f-026b-495e-8b94-6943662fe25e-console-oauth-config\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.973033 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.972941    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/311b3b3f-026b-495e-8b94-6943662fe25e-console-serving-cert\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:06.976907 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:06.976887    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsbcc\" (UniqueName: \"kubernetes.io/projected/311b3b3f-026b-495e-8b94-6943662fe25e-kube-api-access-jsbcc\") pod \"console-579d944d7c-gk9x9\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") " pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:07.082937 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.082903    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:07.113453 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.113000    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zzsl6"]
Jul 31 23:47:07.195202 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.195166    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-js2f5"]
Jul 31 23:47:07.237325 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.237290    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mjkm4"]
Jul 31 23:47:07.237490 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:47:07.237438    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec95cdf0_c411_409a_a0e5_274712ef6c00.slice/crio-2a721af7604a66482ce8f3dc31fb5b7efded8c775f0ab099934e782c5921feba WatchSource:0}: Error finding container 2a721af7604a66482ce8f3dc31fb5b7efded8c775f0ab099934e782c5921feba: Status 404 returned error can't find the container with id 2a721af7604a66482ce8f3dc31fb5b7efded8c775f0ab099934e782c5921feba
Jul 31 23:47:07.242653 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:47:07.242631    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824b4cc2_7b16_43c6_9cc0_46759ffdf9b5.slice/crio-54e0017b4bf223f05743d206b871f3e589dbfc337a679c6a44fa71f770602b6d WatchSource:0}: Error finding container 54e0017b4bf223f05743d206b871f3e589dbfc337a679c6a44fa71f770602b6d: Status 404 returned error can't find the container with id 54e0017b4bf223f05743d206b871f3e589dbfc337a679c6a44fa71f770602b6d
Jul 31 23:47:07.285289 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.284595    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:47:07.285289 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.284655    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:47:07.291467 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.291440    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:47:07.311340 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.311270    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-579d944d7c-gk9x9"]
Jul 31 23:47:07.317466 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:47:07.317430    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311b3b3f_026b_495e_8b94_6943662fe25e.slice/crio-b9ca41e2910ab35f6faa76a0ab8d22b4a63cc13de1a6e74d3b0652d6fcdcae1d WatchSource:0}: Error finding container b9ca41e2910ab35f6faa76a0ab8d22b4a63cc13de1a6e74d3b0652d6fcdcae1d: Status 404 returned error can't find the container with id b9ca41e2910ab35f6faa76a0ab8d22b4a63cc13de1a6e74d3b0652d6fcdcae1d
Jul 31 23:47:07.483422 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.483388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mjkm4" event={"ID":"824b4cc2-7b16-43c6-9cc0-46759ffdf9b5","Type":"ContainerStarted","Data":"54e0017b4bf223f05743d206b871f3e589dbfc337a679c6a44fa71f770602b6d"}
Jul 31 23:47:07.485411 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.485378    2578 generic.go:356] "Generic (PLEG): container finished" podID="2e5bad06-e384-45d4-8863-7d2563e0d648" containerID="7fa85d2c0b8208545ca7e819f6f0d065cf4827984d5199e7308e110b6b51ba12" exitCode=0
Jul 31 23:47:07.485548 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.485462    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75gvm" event={"ID":"2e5bad06-e384-45d4-8863-7d2563e0d648","Type":"ContainerDied","Data":"7fa85d2c0b8208545ca7e819f6f0d065cf4827984d5199e7308e110b6b51ba12"}
Jul 31 23:47:07.487663 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.487584    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579d944d7c-gk9x9" event={"ID":"311b3b3f-026b-495e-8b94-6943662fe25e","Type":"ContainerStarted","Data":"a4f90216beefe9630060fa82b0d1fd7ed92ea2f3c0d16482bd8ca37b55a8a49c"}
Jul 31 23:47:07.487663 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.487637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579d944d7c-gk9x9" event={"ID":"311b3b3f-026b-495e-8b94-6943662fe25e","Type":"ContainerStarted","Data":"b9ca41e2910ab35f6faa76a0ab8d22b4a63cc13de1a6e74d3b0652d6fcdcae1d"}
Jul 31 23:47:07.489351 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.489327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-js2f5" event={"ID":"ec95cdf0-c411-409a-a0e5-274712ef6c00","Type":"ContainerStarted","Data":"2a721af7604a66482ce8f3dc31fb5b7efded8c775f0ab099934e782c5921feba"}
Jul 31 23:47:07.492532 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.492511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l" event={"ID":"46bbcd5c-1c67-4966-8290-90106568f72b","Type":"ContainerStarted","Data":"713f0a2681e8c16f5a021109640bbb7e1ad6b721d5cdf3cf04ce5c7eda808f7e"}
Jul 31 23:47:07.492629 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.492563    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l" event={"ID":"46bbcd5c-1c67-4966-8290-90106568f72b","Type":"ContainerStarted","Data":"57d1948f7f694a2d599b45388b21165c4edc0040e717ef2ebdd840798a4f083d"}
Jul 31 23:47:07.492693 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.492625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l" event={"ID":"46bbcd5c-1c67-4966-8290-90106568f72b","Type":"ContainerStarted","Data":"1f915ca2bb1e89186167cd49981cf09761bafa52a691c24284fd487e9b030593"}
Jul 31 23:47:07.494010 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.493981    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zzsl6" event={"ID":"ab5415d7-3b45-4861-bfaf-be67c4d16871","Type":"ContainerStarted","Data":"c640a8a5f72d84c239a8b0203e0b912958f4bf1f40bd359e4f93e364caeacbfc"}
Jul 31 23:47:07.500147 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.500126    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:47:07.520830 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.520686    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-579d944d7c-gk9x9" podStartSLOduration=1.520664257 podStartE2EDuration="1.520664257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:07.518879842 +0000 UTC m=+66.878770884" watchObservedRunningTime="2026-07-31 23:47:07.520664257 +0000 UTC m=+66.880555300"
Jul 31 23:47:07.563067 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:07.563009    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-r4f5l" podStartSLOduration=2.5710230960000002 podStartE2EDuration="4.562984214s" totalImagesPullingTime="1.991961118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:07.562875572 +0000 UTC m=+66.922766633" watchObservedRunningTime="2026-07-31 23:47:07.562984214 +0000 UTC m=+66.922875255"
Jul 31 23:47:08.503848 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.503754    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-js2f5" event={"ID":"ec95cdf0-c411-409a-a0e5-274712ef6c00","Type":"ContainerStarted","Data":"bb8aa1e556cee9c9a9a561fea6404f7c1518ba6ff75b3fd8c4c8d9d99a1336a4"}
Jul 31 23:47:08.503848 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.503816    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-js2f5" event={"ID":"ec95cdf0-c411-409a-a0e5-274712ef6c00","Type":"ContainerStarted","Data":"39268b617f3145f2672ff72ab37825ab3c5b529ebcbbe5bc7bf1de207edda78a"}
Jul 31 23:47:08.507930 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.507901    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75gvm" event={"ID":"2e5bad06-e384-45d4-8863-7d2563e0d648","Type":"ContainerStarted","Data":"d1f8d9426ed9ca8a30549584ee505b2c33560ea37405757642c0389e6ce2a62b"}
Jul 31 23:47:08.508082 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.507952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75gvm" event={"ID":"2e5bad06-e384-45d4-8863-7d2563e0d648","Type":"ContainerStarted","Data":"6102abe59601abcd50e44657e10e98cdf3894c376dc3d26bed77a5dbf1949965"}
Jul 31 23:47:08.525063 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.525013    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-js2f5" podStartSLOduration=66.525546446 podStartE2EDuration="1m7.524995816s" totalImagesPullingTime="999.44937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:08.524325484 +0000 UTC m=+67.884216552" watchObservedRunningTime="2026-07-31 23:47:08.524995816 +0000 UTC m=+67.884886874"
Jul 31 23:47:08.580734 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.580654    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-75gvm" podStartSLOduration=3.310942276 podStartE2EDuration="5.580633114s" totalImagesPullingTime="2.099065885s" totalInitContainerRuntime="170.624953ms" isStatefulPod=true podCreationTimestamp="2026-07-31 23:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:08.545136211 +0000 UTC m=+67.905027269" watchObservedRunningTime="2026-07-31 23:47:08.580633114 +0000 UTC m=+67.940524164"
Jul 31 23:47:08.582208 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.581165    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-l7z68"]
Jul 31 23:47:08.584582 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.584556    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-l7z68"
Jul 31 23:47:08.588396 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.588007    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-q9864\""
Jul 31 23:47:08.588396 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.588084    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 31 23:47:08.597441 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.597417    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-l7z68"]
Jul 31 23:47:08.686036 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.686000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbglc\" (UniqueName: \"kubernetes.io/projected/311b3c95-8d5d-485a-8037-d473da6787d4-kube-api-access-kbglc\") pod \"monitoring-plugin-6dffbbd67-l7z68\" (UID: \"311b3c95-8d5d-485a-8037-d473da6787d4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-l7z68"
Jul 31 23:47:08.686221 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.686065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/311b3c95-8d5d-485a-8037-d473da6787d4-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-l7z68\" (UID: \"311b3c95-8d5d-485a-8037-d473da6787d4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-l7z68"
Jul 31 23:47:08.787939 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.787859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbglc\" (UniqueName: \"kubernetes.io/projected/311b3c95-8d5d-485a-8037-d473da6787d4-kube-api-access-kbglc\") pod \"monitoring-plugin-6dffbbd67-l7z68\" (UID: \"311b3c95-8d5d-485a-8037-d473da6787d4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-l7z68"
Jul 31 23:47:08.787939 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.787927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/311b3c95-8d5d-485a-8037-d473da6787d4-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-l7z68\" (UID: \"311b3c95-8d5d-485a-8037-d473da6787d4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-l7z68"
Jul 31 23:47:08.791407 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.791375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/311b3c95-8d5d-485a-8037-d473da6787d4-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-l7z68\" (UID: \"311b3c95-8d5d-485a-8037-d473da6787d4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-l7z68"
Jul 31 23:47:08.796486 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.796451    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbglc\" (UniqueName: \"kubernetes.io/projected/311b3c95-8d5d-485a-8037-d473da6787d4-kube-api-access-kbglc\") pod \"monitoring-plugin-6dffbbd67-l7z68\" (UID: \"311b3c95-8d5d-485a-8037-d473da6787d4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-l7z68"
Jul 31 23:47:08.900281 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.900244    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-l7z68"
Jul 31 23:47:08.998669 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:08.997966    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"]
Jul 31 23:47:09.002887 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.002867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.008393 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.007115    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 31 23:47:09.008393 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.007518    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 31 23:47:09.008393 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.007771    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 31 23:47:09.008393 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.007972    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mv5fx\""
Jul 31 23:47:09.008393 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.008384    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 31 23:47:09.008817 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.008582    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 31 23:47:09.021260 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.021235    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 31 23:47:09.029280 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.029250    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"]
Jul 31 23:47:09.069006 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.068885    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-l7z68"]
Jul 31 23:47:09.073939 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:47:09.073909    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311b3c95_8d5d_485a_8037_d473da6787d4.slice/crio-b5bf57a1b2f01c1a701a7d103cb39e8c963b925cee21f2d32865e8f9a8c1a218 WatchSource:0}: Error finding container b5bf57a1b2f01c1a701a7d103cb39e8c963b925cee21f2d32865e8f9a8c1a218: Status 404 returned error can't find the container with id b5bf57a1b2f01c1a701a7d103cb39e8c963b925cee21f2d32865e8f9a8c1a218
Jul 31 23:47:09.089338 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.089306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070d2df4-7d6c-42ae-ae2f-696260f6162a-serving-certs-ca-bundle\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.089471 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.089361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/070d2df4-7d6c-42ae-ae2f-696260f6162a-metrics-client-ca\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.089471 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.089413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2wpb\" (UniqueName: \"kubernetes.io/projected/070d2df4-7d6c-42ae-ae2f-696260f6162a-kube-api-access-x2wpb\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.089471 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.089441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/070d2df4-7d6c-42ae-ae2f-696260f6162a-secret-telemeter-client\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.089471 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.089468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/070d2df4-7d6c-42ae-ae2f-696260f6162a-federate-client-tls\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.089674 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.089486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/070d2df4-7d6c-42ae-ae2f-696260f6162a-telemeter-client-tls\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.089674 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.089502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/070d2df4-7d6c-42ae-ae2f-696260f6162a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.089674 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.089527    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070d2df4-7d6c-42ae-ae2f-696260f6162a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.190074 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.190038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2wpb\" (UniqueName: \"kubernetes.io/projected/070d2df4-7d6c-42ae-ae2f-696260f6162a-kube-api-access-x2wpb\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.190243 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.190089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/070d2df4-7d6c-42ae-ae2f-696260f6162a-secret-telemeter-client\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.190243 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.190116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/070d2df4-7d6c-42ae-ae2f-696260f6162a-federate-client-tls\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.190243 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.190146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/070d2df4-7d6c-42ae-ae2f-696260f6162a-telemeter-client-tls\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.190243 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.190172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/070d2df4-7d6c-42ae-ae2f-696260f6162a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.190391 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.190248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070d2df4-7d6c-42ae-ae2f-696260f6162a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.190391 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.190346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070d2df4-7d6c-42ae-ae2f-696260f6162a-serving-certs-ca-bundle\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.190391 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.190380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/070d2df4-7d6c-42ae-ae2f-696260f6162a-metrics-client-ca\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.191671 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.191079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/070d2df4-7d6c-42ae-ae2f-696260f6162a-metrics-client-ca\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.191671 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.191474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070d2df4-7d6c-42ae-ae2f-696260f6162a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.191671 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.191614    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070d2df4-7d6c-42ae-ae2f-696260f6162a-serving-certs-ca-bundle\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.193241 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.193212    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/070d2df4-7d6c-42ae-ae2f-696260f6162a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.194090 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.194058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/070d2df4-7d6c-42ae-ae2f-696260f6162a-telemeter-client-tls\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.195185 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.195155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/070d2df4-7d6c-42ae-ae2f-696260f6162a-federate-client-tls\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.195706 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.195671    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/070d2df4-7d6c-42ae-ae2f-696260f6162a-secret-telemeter-client\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.199276 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.199251    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2wpb\" (UniqueName: \"kubernetes.io/projected/070d2df4-7d6c-42ae-ae2f-696260f6162a-kube-api-access-x2wpb\") pod \"telemeter-client-7b6dd4b7c9-g59jt\" (UID: \"070d2df4-7d6c-42ae-ae2f-696260f6162a\") " pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.319145 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.319059    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"
Jul 31 23:47:09.439973 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.439944    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5nwm2"
Jul 31 23:47:09.517470 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.516966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-l7z68" event={"ID":"311b3c95-8d5d-485a-8037-d473da6787d4","Type":"ContainerStarted","Data":"b5bf57a1b2f01c1a701a7d103cb39e8c963b925cee21f2d32865e8f9a8c1a218"}
Jul 31 23:47:09.526144 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:09.526109    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt"]
Jul 31 23:47:12.254273 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:47:12.254237    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070d2df4_7d6c_42ae_ae2f_696260f6162a.slice/crio-4be1ecab20ba98b285b88baa702a44d791e7a12fc2b926012347def39715acaa WatchSource:0}: Error finding container 4be1ecab20ba98b285b88baa702a44d791e7a12fc2b926012347def39715acaa: Status 404 returned error can't find the container with id 4be1ecab20ba98b285b88baa702a44d791e7a12fc2b926012347def39715acaa
Jul 31 23:47:12.533445 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:12.533362    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt" event={"ID":"070d2df4-7d6c-42ae-ae2f-696260f6162a","Type":"ContainerStarted","Data":"4be1ecab20ba98b285b88baa702a44d791e7a12fc2b926012347def39715acaa"}
Jul 31 23:47:13.539442 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.539403    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zzsl6" event={"ID":"ab5415d7-3b45-4861-bfaf-be67c4d16871","Type":"ContainerStarted","Data":"b0536b9c25a866f15307202fcd3b5d67a4151de9f2430911bc86ccd242079a9c"}
Jul 31 23:47:13.541615 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.541550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-l7z68" event={"ID":"311b3c95-8d5d-485a-8037-d473da6787d4","Type":"ContainerStarted","Data":"bcb844c5b371b26615407b2923d2b8ff5c06cb9a9cd7d29d74e3375e94a21f57"}
Jul 31 23:47:13.542038 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.541993    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-l7z68"
Jul 31 23:47:13.544127 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.544024    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mjkm4" event={"ID":"824b4cc2-7b16-43c6-9cc0-46759ffdf9b5","Type":"ContainerStarted","Data":"13cf0fc6b20f1dd88df9a973da15ccb8a872b3d18eb60e152b7ae2b1f3e220d2"}
Jul 31 23:47:13.544127 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.544081    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:47:13.548980 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.548958    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-l7z68"
Jul 31 23:47:13.556581 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.556544    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zzsl6" podStartSLOduration=67.022475226 podStartE2EDuration="1m12.556530438s" totalImagesPullingTime="5.534055212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:13.554876574 +0000 UTC m=+72.914767615" watchObservedRunningTime="2026-07-31 23:47:13.556530438 +0000 UTC m=+72.916421478"
Jul 31 23:47:13.575627 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.575556    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-l7z68" podStartSLOduration=1.9218186 podStartE2EDuration="5.575540233s" totalImagesPullingTime="3.653721633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:13.575002483 +0000 UTC m=+72.934893549" watchObservedRunningTime="2026-07-31 23:47:13.575540233 +0000 UTC m=+72.935431274"
Jul 31 23:47:13.592735 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.592680    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mjkm4" podStartSLOduration=67.173954471 podStartE2EDuration="1m12.592666086s" totalImagesPullingTime="5.418711615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:13.591021103 +0000 UTC m=+72.950912144" watchObservedRunningTime="2026-07-31 23:47:13.592666086 +0000 UTC m=+72.952557180"
Jul 31 23:47:13.763861 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.763827    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-579d944d7c-gk9x9"]
Jul 31 23:47:13.793848 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.793775    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6878c4c4cd-hxb9x"]
Jul 31 23:47:13.797244 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.797215    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:13.807760 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.807732    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6878c4c4cd-hxb9x"]
Jul 31 23:47:13.937934 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.937893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28af75a9-111c-44f9-9180-d3dbbb18f902-console-oauth-config\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:13.937934 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.937934    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-console-config\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:13.938174 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.937984    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28af75a9-111c-44f9-9180-d3dbbb18f902-console-serving-cert\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:13.938174 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.938010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-trusted-ca-bundle\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:13.938174 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.938028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-service-ca\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:13.938174 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.938046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-oauth-serving-cert\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:13.938174 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:13.938081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9npfc\" (UniqueName: \"kubernetes.io/projected/28af75a9-111c-44f9-9180-d3dbbb18f902-kube-api-access-9npfc\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:14.039516 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.039478    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28af75a9-111c-44f9-9180-d3dbbb18f902-console-oauth-config\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:14.039516 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.039516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-console-config\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:14.039792 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.039746    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28af75a9-111c-44f9-9180-d3dbbb18f902-console-serving-cert\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:14.039792 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.039784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-trusted-ca-bundle\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:14.039917 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.039810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-service-ca\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:14.039917 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.039833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-oauth-serving-cert\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:14.039917 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.039869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9npfc\" (UniqueName: \"kubernetes.io/projected/28af75a9-111c-44f9-9180-d3dbbb18f902-kube-api-access-9npfc\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:14.040398 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.040368    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-console-config\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:14.040633 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.040605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-service-ca\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:14.040876 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.040849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-oauth-serving-cert\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:14.040876 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.040867    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-trusted-ca-bundle\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:14.042490 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.042465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28af75a9-111c-44f9-9180-d3dbbb18f902-console-serving-cert\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:14.042588 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.042468    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28af75a9-111c-44f9-9180-d3dbbb18f902-console-oauth-config\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:14.047339 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.047283    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9npfc\" (UniqueName: \"kubernetes.io/projected/28af75a9-111c-44f9-9180-d3dbbb18f902-kube-api-access-9npfc\") pod \"console-6878c4c4cd-hxb9x\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") " pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:14.111395 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.111359    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:14.402230 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.402207    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6878c4c4cd-hxb9x"]
Jul 31 23:47:14.406043 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:47:14.406016    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28af75a9_111c_44f9_9180_d3dbbb18f902.slice/crio-d910adb333cac185c275c1d02921a0e70d1402c0f39eb498c6650cba382530ce WatchSource:0}: Error finding container d910adb333cac185c275c1d02921a0e70d1402c0f39eb498c6650cba382530ce: Status 404 returned error can't find the container with id d910adb333cac185c275c1d02921a0e70d1402c0f39eb498c6650cba382530ce
Jul 31 23:47:14.550326 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.549825    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6878c4c4cd-hxb9x" event={"ID":"28af75a9-111c-44f9-9180-d3dbbb18f902","Type":"ContainerStarted","Data":"183a951931e81fb547869997c9fe6f8a1d55023fc43477a97681c40abdc8bdb9"}
Jul 31 23:47:14.550326 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.549886    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6878c4c4cd-hxb9x" event={"ID":"28af75a9-111c-44f9-9180-d3dbbb18f902","Type":"ContainerStarted","Data":"d910adb333cac185c275c1d02921a0e70d1402c0f39eb498c6650cba382530ce"}
Jul 31 23:47:14.552364 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.552339    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt" event={"ID":"070d2df4-7d6c-42ae-ae2f-696260f6162a","Type":"ContainerStarted","Data":"8ada7e1e97db7175d6f0a3ad7dd17e114fe1b6032651ee721393728dfbadfd07"}
Jul 31 23:47:14.565258 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:14.565205    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6878c4c4cd-hxb9x" podStartSLOduration=1.565178824 podStartE2EDuration="1.565178824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:14.565073531 +0000 UTC m=+73.924964586" watchObservedRunningTime="2026-07-31 23:47:14.565178824 +0000 UTC m=+73.925069870"
Jul 31 23:47:15.559910 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:15.559867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt" event={"ID":"070d2df4-7d6c-42ae-ae2f-696260f6162a","Type":"ContainerStarted","Data":"ca7054b1d2b94dc1f003624d1d8fe363f49ad703fe1cf3e0331acf3e84f97e11"}
Jul 31 23:47:15.560368 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:15.559931    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt" event={"ID":"070d2df4-7d6c-42ae-ae2f-696260f6162a","Type":"ContainerStarted","Data":"9ca36daf94aad36a00fad4d3daa9e0aa573a14ee0b8da33c97f20329c39ff408"}
Jul 31 23:47:15.590738 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:15.590604    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7b6dd4b7c9-g59jt" podStartSLOduration=4.648931005 podStartE2EDuration="7.590582132s" totalImagesPullingTime="2.941651127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:15.589147967 +0000 UTC m=+74.949039011" watchObservedRunningTime="2026-07-31 23:47:15.590582132 +0000 UTC m=+74.950473172"
Jul 31 23:47:17.084168 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.084130    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:17.218781 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.218747    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6878c4c4cd-hxb9x"]
Jul 31 23:47:17.245209 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.245176    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74fc4ccf46-7qdvg"]
Jul 31 23:47:17.248638 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.248616    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.270040 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.270009    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74fc4ccf46-7qdvg"]
Jul 31 23:47:17.371054 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.370956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-oauth-serving-cert\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.371054 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.370990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-console-config\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.371054 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.371013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ab71d8a-473b-4440-be63-d05dfe5edbac-console-serving-cert\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.371054 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.371031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-trusted-ca-bundle\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.371358 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.371169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ab71d8a-473b-4440-be63-d05dfe5edbac-console-oauth-config\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.371358 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.371204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfqlg\" (UniqueName: \"kubernetes.io/projected/7ab71d8a-473b-4440-be63-d05dfe5edbac-kube-api-access-dfqlg\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.371358 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.371233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-service-ca\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.472277 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.472238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-service-ca\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.472277 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.472278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-oauth-serving-cert\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.472515 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.472304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-console-config\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.472515 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.472328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ab71d8a-473b-4440-be63-d05dfe5edbac-console-serving-cert\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.472515 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.472348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-trusted-ca-bundle\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.472515 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.472444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ab71d8a-473b-4440-be63-d05dfe5edbac-console-oauth-config\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.472515 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.472466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfqlg\" (UniqueName: \"kubernetes.io/projected/7ab71d8a-473b-4440-be63-d05dfe5edbac-kube-api-access-dfqlg\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.473074 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.473047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-service-ca\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.473186 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.473113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-oauth-serving-cert\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.473186 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.473164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-console-config\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.473453 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.473263    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-trusted-ca-bundle\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.475001 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.474978    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ab71d8a-473b-4440-be63-d05dfe5edbac-console-serving-cert\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.475111 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.475011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ab71d8a-473b-4440-be63-d05dfe5edbac-console-oauth-config\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.480281 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.480259    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfqlg\" (UniqueName: \"kubernetes.io/projected/7ab71d8a-473b-4440-be63-d05dfe5edbac-kube-api-access-dfqlg\") pod \"console-74fc4ccf46-7qdvg\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") " pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.559442 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.559407    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:17.700041 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:17.700009    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74fc4ccf46-7qdvg"]
Jul 31 23:47:17.703468 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:47:17.703445    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab71d8a_473b_4440_be63_d05dfe5edbac.slice/crio-250792bc12d36a5d8eee23f15bd7851dc7bd0da37f201421f8c8b35dc4b9faa1 WatchSource:0}: Error finding container 250792bc12d36a5d8eee23f15bd7851dc7bd0da37f201421f8c8b35dc4b9faa1: Status 404 returned error can't find the container with id 250792bc12d36a5d8eee23f15bd7851dc7bd0da37f201421f8c8b35dc4b9faa1
Jul 31 23:47:18.575884 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:18.575852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fc4ccf46-7qdvg" event={"ID":"7ab71d8a-473b-4440-be63-d05dfe5edbac","Type":"ContainerStarted","Data":"361c23f386310127a43214b29b96485200e2250b78e0deeffb804760dfb5f090"}
Jul 31 23:47:18.576238 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:18.575899    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fc4ccf46-7qdvg" event={"ID":"7ab71d8a-473b-4440-be63-d05dfe5edbac","Type":"ContainerStarted","Data":"250792bc12d36a5d8eee23f15bd7851dc7bd0da37f201421f8c8b35dc4b9faa1"}
Jul 31 23:47:18.596331 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:18.596284    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74fc4ccf46-7qdvg" podStartSLOduration=1.596267309 podStartE2EDuration="1.596267309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:18.595595657 +0000 UTC m=+77.955486697" watchObservedRunningTime="2026-07-31 23:47:18.596267309 +0000 UTC m=+77.956158667"
Jul 31 23:47:24.111809 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:24.111684    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:27.559641 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:27.559607    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:27.560055 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:27.559685    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:27.564345 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:27.564317    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:27.613929 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:27.613901    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:47:27.673108 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:27.673079    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bcf7bf76f-s6hlw"]
Jul 31 23:47:38.783603 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:38.783537    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-579d944d7c-gk9x9" podUID="311b3b3f-026b-495e-8b94-6943662fe25e" containerName="console" containerID="cri-o://a4f90216beefe9630060fa82b0d1fd7ed92ea2f3c0d16482bd8ca37b55a8a49c" gracePeriod=15
Jul 31 23:47:39.044950 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.044884    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:39.055824 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.055801    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-trusted-ca-bundle\") pod \"311b3b3f-026b-495e-8b94-6943662fe25e\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") "
Jul 31 23:47:39.055961 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.055833    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-service-ca\") pod \"311b3b3f-026b-495e-8b94-6943662fe25e\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") "
Jul 31 23:47:39.055961 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.055850    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsbcc\" (UniqueName: \"kubernetes.io/projected/311b3b3f-026b-495e-8b94-6943662fe25e-kube-api-access-jsbcc\") pod \"311b3b3f-026b-495e-8b94-6943662fe25e\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") "
Jul 31 23:47:39.055961 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.055879    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-console-config\") pod \"311b3b3f-026b-495e-8b94-6943662fe25e\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") "
Jul 31 23:47:39.055961 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.055905    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/311b3b3f-026b-495e-8b94-6943662fe25e-console-serving-cert\") pod \"311b3b3f-026b-495e-8b94-6943662fe25e\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") "
Jul 31 23:47:39.055961 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.055940    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-oauth-serving-cert\") pod \"311b3b3f-026b-495e-8b94-6943662fe25e\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") "
Jul 31 23:47:39.056227 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.056039    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/311b3b3f-026b-495e-8b94-6943662fe25e-console-oauth-config\") pod \"311b3b3f-026b-495e-8b94-6943662fe25e\" (UID: \"311b3b3f-026b-495e-8b94-6943662fe25e\") "
Jul 31 23:47:39.056361 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.056340    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "311b3b3f-026b-495e-8b94-6943662fe25e" (UID: "311b3b3f-026b-495e-8b94-6943662fe25e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:47:39.056424 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.056364    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-service-ca" (OuterVolumeSpecName: "service-ca") pod "311b3b3f-026b-495e-8b94-6943662fe25e" (UID: "311b3b3f-026b-495e-8b94-6943662fe25e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:47:39.056424 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.056359    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-console-config" (OuterVolumeSpecName: "console-config") pod "311b3b3f-026b-495e-8b94-6943662fe25e" (UID: "311b3b3f-026b-495e-8b94-6943662fe25e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:47:39.056534 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.056457    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "311b3b3f-026b-495e-8b94-6943662fe25e" (UID: "311b3b3f-026b-495e-8b94-6943662fe25e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:47:39.058446 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.058412    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311b3b3f-026b-495e-8b94-6943662fe25e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "311b3b3f-026b-495e-8b94-6943662fe25e" (UID: "311b3b3f-026b-495e-8b94-6943662fe25e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:47:39.058539 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.058476    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311b3b3f-026b-495e-8b94-6943662fe25e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "311b3b3f-026b-495e-8b94-6943662fe25e" (UID: "311b3b3f-026b-495e-8b94-6943662fe25e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:47:39.058539 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.058501    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311b3b3f-026b-495e-8b94-6943662fe25e-kube-api-access-jsbcc" (OuterVolumeSpecName: "kube-api-access-jsbcc") pod "311b3b3f-026b-495e-8b94-6943662fe25e" (UID: "311b3b3f-026b-495e-8b94-6943662fe25e"). InnerVolumeSpecName "kube-api-access-jsbcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:47:39.157221 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.157182    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/311b3b3f-026b-495e-8b94-6943662fe25e-console-oauth-config\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:39.157221 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.157215    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-trusted-ca-bundle\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:39.157221 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.157226    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-service-ca\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:39.157447 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.157235    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsbcc\" (UniqueName: \"kubernetes.io/projected/311b3b3f-026b-495e-8b94-6943662fe25e-kube-api-access-jsbcc\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:39.157447 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.157244    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-console-config\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:39.157447 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.157252    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/311b3b3f-026b-495e-8b94-6943662fe25e-console-serving-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:39.157447 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.157260    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/311b3b3f-026b-495e-8b94-6943662fe25e-oauth-serving-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:39.654813 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.654779    2578 generic.go:356] "Generic (PLEG): container finished" podID="311b3b3f-026b-495e-8b94-6943662fe25e" containerID="a4f90216beefe9630060fa82b0d1fd7ed92ea2f3c0d16482bd8ca37b55a8a49c" exitCode=0
Jul 31 23:47:39.655014 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.654823    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579d944d7c-gk9x9"
Jul 31 23:47:39.655014 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.654865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579d944d7c-gk9x9" event={"ID":"311b3b3f-026b-495e-8b94-6943662fe25e","Type":"ContainerDied","Data":"a4f90216beefe9630060fa82b0d1fd7ed92ea2f3c0d16482bd8ca37b55a8a49c"}
Jul 31 23:47:39.655014 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.654920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579d944d7c-gk9x9" event={"ID":"311b3b3f-026b-495e-8b94-6943662fe25e","Type":"ContainerDied","Data":"b9ca41e2910ab35f6faa76a0ab8d22b4a63cc13de1a6e74d3b0652d6fcdcae1d"}
Jul 31 23:47:39.655014 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.654948    2578 scope.go:117] "RemoveContainer" containerID="a4f90216beefe9630060fa82b0d1fd7ed92ea2f3c0d16482bd8ca37b55a8a49c"
Jul 31 23:47:39.663260 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.663246    2578 scope.go:117] "RemoveContainer" containerID="a4f90216beefe9630060fa82b0d1fd7ed92ea2f3c0d16482bd8ca37b55a8a49c"
Jul 31 23:47:39.663511 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:47:39.663492    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f90216beefe9630060fa82b0d1fd7ed92ea2f3c0d16482bd8ca37b55a8a49c\": container with ID starting with a4f90216beefe9630060fa82b0d1fd7ed92ea2f3c0d16482bd8ca37b55a8a49c not found: ID does not exist" containerID="a4f90216beefe9630060fa82b0d1fd7ed92ea2f3c0d16482bd8ca37b55a8a49c"
Jul 31 23:47:39.663555 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.663519    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f90216beefe9630060fa82b0d1fd7ed92ea2f3c0d16482bd8ca37b55a8a49c"} err="failed to get container status \"a4f90216beefe9630060fa82b0d1fd7ed92ea2f3c0d16482bd8ca37b55a8a49c\": rpc error: code = NotFound desc = could not find container \"a4f90216beefe9630060fa82b0d1fd7ed92ea2f3c0d16482bd8ca37b55a8a49c\": container with ID starting with a4f90216beefe9630060fa82b0d1fd7ed92ea2f3c0d16482bd8ca37b55a8a49c not found: ID does not exist"
Jul 31 23:47:39.669001 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.668983    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-579d944d7c-gk9x9"]
Jul 31 23:47:39.673046 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:39.673027    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-579d944d7c-gk9x9"]
Jul 31 23:47:41.184098 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:41.184068    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="311b3b3f-026b-495e-8b94-6943662fe25e" path="/var/lib/kubelet/pods/311b3b3f-026b-495e-8b94-6943662fe25e/volumes"
Jul 31 23:47:42.242102 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.242063    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6878c4c4cd-hxb9x" podUID="28af75a9-111c-44f9-9180-d3dbbb18f902" containerName="console" containerID="cri-o://183a951931e81fb547869997c9fe6f8a1d55023fc43477a97681c40abdc8bdb9" gracePeriod=15
Jul 31 23:47:42.484463 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.484438    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:42.582568 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.582469    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28af75a9-111c-44f9-9180-d3dbbb18f902-console-oauth-config\") pod \"28af75a9-111c-44f9-9180-d3dbbb18f902\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") "
Jul 31 23:47:42.582568 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.582547    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9npfc\" (UniqueName: \"kubernetes.io/projected/28af75a9-111c-44f9-9180-d3dbbb18f902-kube-api-access-9npfc\") pod \"28af75a9-111c-44f9-9180-d3dbbb18f902\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") "
Jul 31 23:47:42.582800 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.582582    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-service-ca\") pod \"28af75a9-111c-44f9-9180-d3dbbb18f902\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") "
Jul 31 23:47:42.582800 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.582609    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-oauth-serving-cert\") pod \"28af75a9-111c-44f9-9180-d3dbbb18f902\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") "
Jul 31 23:47:42.582800 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.582661    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-trusted-ca-bundle\") pod \"28af75a9-111c-44f9-9180-d3dbbb18f902\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") "
Jul 31 23:47:42.582800 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.582703    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28af75a9-111c-44f9-9180-d3dbbb18f902-console-serving-cert\") pod \"28af75a9-111c-44f9-9180-d3dbbb18f902\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") "
Jul 31 23:47:42.582800 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.582754    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-console-config\") pod \"28af75a9-111c-44f9-9180-d3dbbb18f902\" (UID: \"28af75a9-111c-44f9-9180-d3dbbb18f902\") "
Jul 31 23:47:42.583145 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.583080    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-service-ca" (OuterVolumeSpecName: "service-ca") pod "28af75a9-111c-44f9-9180-d3dbbb18f902" (UID: "28af75a9-111c-44f9-9180-d3dbbb18f902"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:47:42.583436 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.583169    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "28af75a9-111c-44f9-9180-d3dbbb18f902" (UID: "28af75a9-111c-44f9-9180-d3dbbb18f902"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:47:42.583436 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.583182    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "28af75a9-111c-44f9-9180-d3dbbb18f902" (UID: "28af75a9-111c-44f9-9180-d3dbbb18f902"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:47:42.583436 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.583266    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-console-config" (OuterVolumeSpecName: "console-config") pod "28af75a9-111c-44f9-9180-d3dbbb18f902" (UID: "28af75a9-111c-44f9-9180-d3dbbb18f902"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:47:42.584875 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.584853    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28af75a9-111c-44f9-9180-d3dbbb18f902-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "28af75a9-111c-44f9-9180-d3dbbb18f902" (UID: "28af75a9-111c-44f9-9180-d3dbbb18f902"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:47:42.584955 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.584877    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28af75a9-111c-44f9-9180-d3dbbb18f902-kube-api-access-9npfc" (OuterVolumeSpecName: "kube-api-access-9npfc") pod "28af75a9-111c-44f9-9180-d3dbbb18f902" (UID: "28af75a9-111c-44f9-9180-d3dbbb18f902"). InnerVolumeSpecName "kube-api-access-9npfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:47:42.585039 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.585025    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28af75a9-111c-44f9-9180-d3dbbb18f902-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "28af75a9-111c-44f9-9180-d3dbbb18f902" (UID: "28af75a9-111c-44f9-9180-d3dbbb18f902"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:47:42.668768 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.668740    2578 generic.go:356] "Generic (PLEG): container finished" podID="28af75a9-111c-44f9-9180-d3dbbb18f902" containerID="183a951931e81fb547869997c9fe6f8a1d55023fc43477a97681c40abdc8bdb9" exitCode=0
Jul 31 23:47:42.668933 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.668824    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6878c4c4cd-hxb9x" event={"ID":"28af75a9-111c-44f9-9180-d3dbbb18f902","Type":"ContainerDied","Data":"183a951931e81fb547869997c9fe6f8a1d55023fc43477a97681c40abdc8bdb9"}
Jul 31 23:47:42.668933 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.668840    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6878c4c4cd-hxb9x"
Jul 31 23:47:42.668933 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.668865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6878c4c4cd-hxb9x" event={"ID":"28af75a9-111c-44f9-9180-d3dbbb18f902","Type":"ContainerDied","Data":"d910adb333cac185c275c1d02921a0e70d1402c0f39eb498c6650cba382530ce"}
Jul 31 23:47:42.668933 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.668891    2578 scope.go:117] "RemoveContainer" containerID="183a951931e81fb547869997c9fe6f8a1d55023fc43477a97681c40abdc8bdb9"
Jul 31 23:47:42.678070 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.678051    2578 scope.go:117] "RemoveContainer" containerID="183a951931e81fb547869997c9fe6f8a1d55023fc43477a97681c40abdc8bdb9"
Jul 31 23:47:42.678361 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:47:42.678341    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"183a951931e81fb547869997c9fe6f8a1d55023fc43477a97681c40abdc8bdb9\": container with ID starting with 183a951931e81fb547869997c9fe6f8a1d55023fc43477a97681c40abdc8bdb9 not found: ID does not exist" containerID="183a951931e81fb547869997c9fe6f8a1d55023fc43477a97681c40abdc8bdb9"
Jul 31 23:47:42.678416 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.678370    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"183a951931e81fb547869997c9fe6f8a1d55023fc43477a97681c40abdc8bdb9"} err="failed to get container status \"183a951931e81fb547869997c9fe6f8a1d55023fc43477a97681c40abdc8bdb9\": rpc error: code = NotFound desc = could not find container \"183a951931e81fb547869997c9fe6f8a1d55023fc43477a97681c40abdc8bdb9\": container with ID starting with 183a951931e81fb547869997c9fe6f8a1d55023fc43477a97681c40abdc8bdb9 not found: ID does not exist"
Jul 31 23:47:42.683731 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.683695    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-trusted-ca-bundle\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:42.683891 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.683742    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28af75a9-111c-44f9-9180-d3dbbb18f902-console-serving-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:42.683891 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.683756    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-console-config\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:42.683891 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.683768    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28af75a9-111c-44f9-9180-d3dbbb18f902-console-oauth-config\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:42.683891 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.683779    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9npfc\" (UniqueName: \"kubernetes.io/projected/28af75a9-111c-44f9-9180-d3dbbb18f902-kube-api-access-9npfc\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:42.683891 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.683790    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-service-ca\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:42.683891 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.683802    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28af75a9-111c-44f9-9180-d3dbbb18f902-oauth-serving-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:42.693333 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.693310    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6878c4c4cd-hxb9x"]
Jul 31 23:47:42.699361 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:42.699340    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6878c4c4cd-hxb9x"]
Jul 31 23:47:43.184533 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:43.184492    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28af75a9-111c-44f9-9180-d3dbbb18f902" path="/var/lib/kubelet/pods/28af75a9-111c-44f9-9180-d3dbbb18f902/volumes"
Jul 31 23:47:44.554661 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:44.554635    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mjkm4"
Jul 31 23:47:52.694907 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:52.694842    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bcf7bf76f-s6hlw" podUID="b4f15005-6696-47b0-9a66-15e23fbe88e2" containerName="console" containerID="cri-o://1496d35c917fa5a3af7551dce3ce79204dc2d4730ffcb5d5626bb5452c6e90cf" gracePeriod=15
Jul 31 23:47:52.934478 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:52.934454    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:47:53.056925 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.056830    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4f15005-6696-47b0-9a66-15e23fbe88e2-oauth-serving-cert\") pod \"b4f15005-6696-47b0-9a66-15e23fbe88e2\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") "
Jul 31 23:47:53.056925 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.056878    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtl27\" (UniqueName: \"kubernetes.io/projected/b4f15005-6696-47b0-9a66-15e23fbe88e2-kube-api-access-mtl27\") pod \"b4f15005-6696-47b0-9a66-15e23fbe88e2\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") "
Jul 31 23:47:53.056925 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.056896    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f15005-6696-47b0-9a66-15e23fbe88e2-service-ca\") pod \"b4f15005-6696-47b0-9a66-15e23fbe88e2\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") "
Jul 31 23:47:53.056925 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.056913    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4f15005-6696-47b0-9a66-15e23fbe88e2-console-oauth-config\") pod \"b4f15005-6696-47b0-9a66-15e23fbe88e2\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") "
Jul 31 23:47:53.057267 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.057051    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4f15005-6696-47b0-9a66-15e23fbe88e2-console-config\") pod \"b4f15005-6696-47b0-9a66-15e23fbe88e2\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") "
Jul 31 23:47:53.057267 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.057136    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4f15005-6696-47b0-9a66-15e23fbe88e2-console-serving-cert\") pod \"b4f15005-6696-47b0-9a66-15e23fbe88e2\" (UID: \"b4f15005-6696-47b0-9a66-15e23fbe88e2\") "
Jul 31 23:47:53.057363 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.057314    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f15005-6696-47b0-9a66-15e23fbe88e2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b4f15005-6696-47b0-9a66-15e23fbe88e2" (UID: "b4f15005-6696-47b0-9a66-15e23fbe88e2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:47:53.057363 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.057318    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f15005-6696-47b0-9a66-15e23fbe88e2-service-ca" (OuterVolumeSpecName: "service-ca") pod "b4f15005-6696-47b0-9a66-15e23fbe88e2" (UID: "b4f15005-6696-47b0-9a66-15e23fbe88e2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:47:53.057441 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.057391    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f15005-6696-47b0-9a66-15e23fbe88e2-console-config" (OuterVolumeSpecName: "console-config") pod "b4f15005-6696-47b0-9a66-15e23fbe88e2" (UID: "b4f15005-6696-47b0-9a66-15e23fbe88e2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:47:53.059314 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.059288    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f15005-6696-47b0-9a66-15e23fbe88e2-kube-api-access-mtl27" (OuterVolumeSpecName: "kube-api-access-mtl27") pod "b4f15005-6696-47b0-9a66-15e23fbe88e2" (UID: "b4f15005-6696-47b0-9a66-15e23fbe88e2"). InnerVolumeSpecName "kube-api-access-mtl27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:47:53.059314 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.059288    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f15005-6696-47b0-9a66-15e23fbe88e2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b4f15005-6696-47b0-9a66-15e23fbe88e2" (UID: "b4f15005-6696-47b0-9a66-15e23fbe88e2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:47:53.059456 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.059318    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f15005-6696-47b0-9a66-15e23fbe88e2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b4f15005-6696-47b0-9a66-15e23fbe88e2" (UID: "b4f15005-6696-47b0-9a66-15e23fbe88e2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:47:53.158377 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.158338    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4f15005-6696-47b0-9a66-15e23fbe88e2-console-serving-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:53.158377 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.158367    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4f15005-6696-47b0-9a66-15e23fbe88e2-oauth-serving-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:53.158377 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.158376    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtl27\" (UniqueName: \"kubernetes.io/projected/b4f15005-6696-47b0-9a66-15e23fbe88e2-kube-api-access-mtl27\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:53.158377 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.158385    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f15005-6696-47b0-9a66-15e23fbe88e2-service-ca\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:53.158625 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.158394    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4f15005-6696-47b0-9a66-15e23fbe88e2-console-oauth-config\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:53.158625 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.158402    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4f15005-6696-47b0-9a66-15e23fbe88e2-console-config\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:47:53.713072 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.713037    2578 generic.go:356] "Generic (PLEG): container finished" podID="b4f15005-6696-47b0-9a66-15e23fbe88e2" containerID="1496d35c917fa5a3af7551dce3ce79204dc2d4730ffcb5d5626bb5452c6e90cf" exitCode=0
Jul 31 23:47:53.713458 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.713090    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bcf7bf76f-s6hlw"
Jul 31 23:47:53.713458 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.713117    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bcf7bf76f-s6hlw" event={"ID":"b4f15005-6696-47b0-9a66-15e23fbe88e2","Type":"ContainerDied","Data":"1496d35c917fa5a3af7551dce3ce79204dc2d4730ffcb5d5626bb5452c6e90cf"}
Jul 31 23:47:53.713458 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.713165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bcf7bf76f-s6hlw" event={"ID":"b4f15005-6696-47b0-9a66-15e23fbe88e2","Type":"ContainerDied","Data":"dc2c5bd5e25dbdf1877b7517e175cd6fadfc39e6b769329152c097b7b01f964a"}
Jul 31 23:47:53.713458 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.713186    2578 scope.go:117] "RemoveContainer" containerID="1496d35c917fa5a3af7551dce3ce79204dc2d4730ffcb5d5626bb5452c6e90cf"
Jul 31 23:47:53.721605 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.721590    2578 scope.go:117] "RemoveContainer" containerID="1496d35c917fa5a3af7551dce3ce79204dc2d4730ffcb5d5626bb5452c6e90cf"
Jul 31 23:47:53.721893 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:47:53.721876    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1496d35c917fa5a3af7551dce3ce79204dc2d4730ffcb5d5626bb5452c6e90cf\": container with ID starting with 1496d35c917fa5a3af7551dce3ce79204dc2d4730ffcb5d5626bb5452c6e90cf not found: ID does not exist" containerID="1496d35c917fa5a3af7551dce3ce79204dc2d4730ffcb5d5626bb5452c6e90cf"
Jul 31 23:47:53.721961 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.721899    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1496d35c917fa5a3af7551dce3ce79204dc2d4730ffcb5d5626bb5452c6e90cf"} err="failed to get container status \"1496d35c917fa5a3af7551dce3ce79204dc2d4730ffcb5d5626bb5452c6e90cf\": rpc error: code = NotFound desc = could not find container \"1496d35c917fa5a3af7551dce3ce79204dc2d4730ffcb5d5626bb5452c6e90cf\": container with ID starting with 1496d35c917fa5a3af7551dce3ce79204dc2d4730ffcb5d5626bb5452c6e90cf not found: ID does not exist"
Jul 31 23:47:53.727263 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.727240    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bcf7bf76f-s6hlw"]
Jul 31 23:47:53.730842 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:53.730823    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bcf7bf76f-s6hlw"]
Jul 31 23:47:55.183739 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:47:55.183692    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4f15005-6696-47b0-9a66-15e23fbe88e2" path="/var/lib/kubelet/pods/b4f15005-6696-47b0-9a66-15e23fbe88e2/volumes"
Jul 31 23:48:25.692604 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.692549    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:48:25.693209 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.692849    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f15005-6696-47b0-9a66-15e23fbe88e2" containerName="console"
Jul 31 23:48:25.693209 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.692860    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f15005-6696-47b0-9a66-15e23fbe88e2" containerName="console"
Jul 31 23:48:25.693209 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.692874    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311b3b3f-026b-495e-8b94-6943662fe25e" containerName="console"
Jul 31 23:48:25.693209 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.692880    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="311b3b3f-026b-495e-8b94-6943662fe25e" containerName="console"
Jul 31 23:48:25.693209 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.692887    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28af75a9-111c-44f9-9180-d3dbbb18f902" containerName="console"
Jul 31 23:48:25.693209 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.692891    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="28af75a9-111c-44f9-9180-d3dbbb18f902" containerName="console"
Jul 31 23:48:25.693209 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.692935    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="28af75a9-111c-44f9-9180-d3dbbb18f902" containerName="console"
Jul 31 23:48:25.693209 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.692941    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4f15005-6696-47b0-9a66-15e23fbe88e2" containerName="console"
Jul 31 23:48:25.693209 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.692947    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="311b3b3f-026b-495e-8b94-6943662fe25e" containerName="console"
Jul 31 23:48:25.696448 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.696421    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.698250 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.698224    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 31 23:48:25.698369 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.698313    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 31 23:48:25.698477 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.698455    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 31 23:48:25.698616 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.698596    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 31 23:48:25.698695 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.698645    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 31 23:48:25.698788 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.698649    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k2kx5\""
Jul 31 23:48:25.698788 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.698596    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 31 23:48:25.698788 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.698775    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 31 23:48:25.698940 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.698650    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 31 23:48:25.704980 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.704956    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 31 23:48:25.714901 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.714878    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:48:25.806735 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.806676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.806903 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.806739    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/184415c8-ee5b-4373-b59d-993997326934-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.806903 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.806776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184415c8-ee5b-4373-b59d-993997326934-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.806903 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.806823    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.806903 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.806842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.807036 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.806900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xthh\" (UniqueName: \"kubernetes.io/projected/184415c8-ee5b-4373-b59d-993997326934-kube-api-access-6xthh\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.807036 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.806930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.807036 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.806970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-config-volume\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.807036 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.806987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-web-config\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.807036 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.807012    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/184415c8-ee5b-4373-b59d-993997326934-tls-assets\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.807036 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.807027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/184415c8-ee5b-4373-b59d-993997326934-config-out\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.807209 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.807042    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/184415c8-ee5b-4373-b59d-993997326934-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.807209 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.807118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.908496 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.908448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/184415c8-ee5b-4373-b59d-993997326934-config-out\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.908496 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.908494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/184415c8-ee5b-4373-b59d-993997326934-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.908805 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.908520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.908805 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.908642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.908805 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.908677    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/184415c8-ee5b-4373-b59d-993997326934-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.908975 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.908800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184415c8-ee5b-4373-b59d-993997326934-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.908975 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.908914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.908975 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.908947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.909123 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.908980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xthh\" (UniqueName: \"kubernetes.io/projected/184415c8-ee5b-4373-b59d-993997326934-kube-api-access-6xthh\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.909123 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.909005    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/184415c8-ee5b-4373-b59d-993997326934-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.909123 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.909009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.909123 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.909077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-config-volume\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.909123 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.909108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-web-config\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.909521 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.909155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/184415c8-ee5b-4373-b59d-993997326934-tls-assets\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.909521 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.909365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/184415c8-ee5b-4373-b59d-993997326934-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.910232 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.910201    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184415c8-ee5b-4373-b59d-993997326934-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.911774 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.911703    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/184415c8-ee5b-4373-b59d-993997326934-config-out\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.911892 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.911868    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.912064 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.912041    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/184415c8-ee5b-4373-b59d-993997326934-tls-assets\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.912206 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.912175    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.912206 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.912193    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-config-volume\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.912608 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.912587    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.912836 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.912821    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.913339 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.913323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-web-config\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.913883 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.913864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/184415c8-ee5b-4373-b59d-993997326934-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.917477 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:25.917457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xthh\" (UniqueName: \"kubernetes.io/projected/184415c8-ee5b-4373-b59d-993997326934-kube-api-access-6xthh\") pod \"alertmanager-main-0\" (UID: \"184415c8-ee5b-4373-b59d-993997326934\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:26.010475 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:26.010374    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:26.146217 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:26.146185    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:48:26.149672 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:48:26.149641    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184415c8_ee5b_4373_b59d_993997326934.slice/crio-07ea08570a21b73f593cb72eeebf4fba18cbe0679ee1e8638d3ca45ab0bbd255 WatchSource:0}: Error finding container 07ea08570a21b73f593cb72eeebf4fba18cbe0679ee1e8638d3ca45ab0bbd255: Status 404 returned error can't find the container with id 07ea08570a21b73f593cb72eeebf4fba18cbe0679ee1e8638d3ca45ab0bbd255
Jul 31 23:48:26.830214 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:26.830182    2578 generic.go:356] "Generic (PLEG): container finished" podID="184415c8-ee5b-4373-b59d-993997326934" containerID="2408cb7134fc5c707b481ef7e60197fc0f2a1bd378bbb8024d7b0324c76d596a" exitCode=0
Jul 31 23:48:26.830691 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:26.830239    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"184415c8-ee5b-4373-b59d-993997326934","Type":"ContainerDied","Data":"2408cb7134fc5c707b481ef7e60197fc0f2a1bd378bbb8024d7b0324c76d596a"}
Jul 31 23:48:26.830691 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:26.830289    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"184415c8-ee5b-4373-b59d-993997326934","Type":"ContainerStarted","Data":"07ea08570a21b73f593cb72eeebf4fba18cbe0679ee1e8638d3ca45ab0bbd255"}
Jul 31 23:48:28.842382 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:28.842342    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"184415c8-ee5b-4373-b59d-993997326934","Type":"ContainerStarted","Data":"13078e9f01154b6580e3df8ac20fa4d5481ab506c27a1a8833958745897c4e58"}
Jul 31 23:48:28.842771 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:28.842435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"184415c8-ee5b-4373-b59d-993997326934","Type":"ContainerStarted","Data":"167be73dea052483a79405c00948b938fd3f62b1331bdd8c4f92be2358365fac"}
Jul 31 23:48:28.842771 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:28.842462    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"184415c8-ee5b-4373-b59d-993997326934","Type":"ContainerStarted","Data":"9321237fd3ccd565e7a42fd7c9407a7b702a11311921f74260fe444403b1fb41"}
Jul 31 23:48:28.842771 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:28.842487    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"184415c8-ee5b-4373-b59d-993997326934","Type":"ContainerStarted","Data":"26226d92a3728d9b5f2dbe0deca5365e1afcc703c68393a9d4c0865c49b3275a"}
Jul 31 23:48:28.842771 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:28.842510    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"184415c8-ee5b-4373-b59d-993997326934","Type":"ContainerStarted","Data":"290405b2f67e267e53050874de855cf01b173c96b2557c0387e1ab4b6c3f9fd0"}
Jul 31 23:48:29.849645 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:29.849562    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"184415c8-ee5b-4373-b59d-993997326934","Type":"ContainerStarted","Data":"4d89c55d8d8c2f60f141865628c7b8c3b135e658416ae9daf5f86ada2b2eae7d"}
Jul 31 23:48:29.877197 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:29.877141    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.509836634 podStartE2EDuration="4.877124171s" totalImagesPullingTime="2.34707451s" totalInitContainerRuntime="20.213027ms" isStatefulPod=false podCreationTimestamp="2026-07-31 23:48:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:48:29.875825462 +0000 UTC m=+149.235716521" watchObservedRunningTime="2026-07-31 23:48:29.877124171 +0000 UTC m=+149.237015212"
Jul 31 23:48:34.497085 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.497054    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fd89469fd-96jjr"]
Jul 31 23:48:34.501488 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.501467    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.509537 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.509516    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd89469fd-96jjr"]
Jul 31 23:48:34.686046 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.686009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61f880b5-8369-40fb-969c-c54fa77f2c59-console-serving-cert\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.686046 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.686048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-service-ca\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.686248 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.686070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-oauth-serving-cert\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.686248 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.686174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45cqr\" (UniqueName: \"kubernetes.io/projected/61f880b5-8369-40fb-969c-c54fa77f2c59-kube-api-access-45cqr\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.686248 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.686207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-trusted-ca-bundle\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.686248 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.686240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61f880b5-8369-40fb-969c-c54fa77f2c59-console-oauth-config\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.686371 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.686281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-console-config\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.787449 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.787351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45cqr\" (UniqueName: \"kubernetes.io/projected/61f880b5-8369-40fb-969c-c54fa77f2c59-kube-api-access-45cqr\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.787449 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.787401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-trusted-ca-bundle\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.787449 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.787424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61f880b5-8369-40fb-969c-c54fa77f2c59-console-oauth-config\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.787681 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.787481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-console-config\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.793389 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.787973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61f880b5-8369-40fb-969c-c54fa77f2c59-console-serving-cert\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.793389 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.788034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-service-ca\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.793389 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.788071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-oauth-serving-cert\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.793389 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.788545    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-console-config\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.793389 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.788801    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-oauth-serving-cert\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.793389 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.789135    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-service-ca\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.793389 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.789333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-trusted-ca-bundle\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.793389 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.790940    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61f880b5-8369-40fb-969c-c54fa77f2c59-console-oauth-config\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.793389 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.791192    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61f880b5-8369-40fb-969c-c54fa77f2c59-console-serving-cert\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.795346 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.795322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45cqr\" (UniqueName: \"kubernetes.io/projected/61f880b5-8369-40fb-969c-c54fa77f2c59-kube-api-access-45cqr\") pod \"console-fd89469fd-96jjr\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") " pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.812293 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.812271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:34.963107 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:34.963076    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd89469fd-96jjr"]
Jul 31 23:48:34.966208 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:48:34.966179    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f880b5_8369_40fb_969c_c54fa77f2c59.slice/crio-eb154ea56ad191e1d763fc9fd8da6605dfc321d30287709c15d7072acd19665e WatchSource:0}: Error finding container eb154ea56ad191e1d763fc9fd8da6605dfc321d30287709c15d7072acd19665e: Status 404 returned error can't find the container with id eb154ea56ad191e1d763fc9fd8da6605dfc321d30287709c15d7072acd19665e
Jul 31 23:48:35.872742 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:35.872695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd89469fd-96jjr" event={"ID":"61f880b5-8369-40fb-969c-c54fa77f2c59","Type":"ContainerStarted","Data":"90bce9c9be40a163c6aa4b165970f6d091b115e2e534fbcc67d5d73d828aba04"}
Jul 31 23:48:35.873175 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:35.872758    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd89469fd-96jjr" event={"ID":"61f880b5-8369-40fb-969c-c54fa77f2c59","Type":"ContainerStarted","Data":"eb154ea56ad191e1d763fc9fd8da6605dfc321d30287709c15d7072acd19665e"}
Jul 31 23:48:35.888241 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:35.888189    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fd89469fd-96jjr" podStartSLOduration=1.888171244 podStartE2EDuration="1.888171244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:48:35.887710482 +0000 UTC m=+155.247601541" watchObservedRunningTime="2026-07-31 23:48:35.888171244 +0000 UTC m=+155.248062285"
Jul 31 23:48:44.813459 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:44.813428    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:44.813879 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:44.813515    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:44.818164 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:44.818141    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:44.907395 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:44.907368    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:48:44.956897 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:48:44.956867    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74fc4ccf46-7qdvg"]
Jul 31 23:49:09.981467 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:09.981355    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74fc4ccf46-7qdvg" podUID="7ab71d8a-473b-4440-be63-d05dfe5edbac" containerName="console" containerID="cri-o://361c23f386310127a43214b29b96485200e2250b78e0deeffb804760dfb5f090" gracePeriod=15
Jul 31 23:49:10.225663 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.225638    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:49:10.352661 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.352562    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfqlg\" (UniqueName: \"kubernetes.io/projected/7ab71d8a-473b-4440-be63-d05dfe5edbac-kube-api-access-dfqlg\") pod \"7ab71d8a-473b-4440-be63-d05dfe5edbac\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") "
Jul 31 23:49:10.352661 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.352608    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-oauth-serving-cert\") pod \"7ab71d8a-473b-4440-be63-d05dfe5edbac\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") "
Jul 31 23:49:10.352661 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.352646    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-console-config\") pod \"7ab71d8a-473b-4440-be63-d05dfe5edbac\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") "
Jul 31 23:49:10.352661 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.352662    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-trusted-ca-bundle\") pod \"7ab71d8a-473b-4440-be63-d05dfe5edbac\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") "
Jul 31 23:49:10.353023 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.352736    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ab71d8a-473b-4440-be63-d05dfe5edbac-console-serving-cert\") pod \"7ab71d8a-473b-4440-be63-d05dfe5edbac\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") "
Jul 31 23:49:10.353023 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.352768    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-service-ca\") pod \"7ab71d8a-473b-4440-be63-d05dfe5edbac\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") "
Jul 31 23:49:10.353023 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.352819    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ab71d8a-473b-4440-be63-d05dfe5edbac-console-oauth-config\") pod \"7ab71d8a-473b-4440-be63-d05dfe5edbac\" (UID: \"7ab71d8a-473b-4440-be63-d05dfe5edbac\") "
Jul 31 23:49:10.353189 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.353153    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-console-config" (OuterVolumeSpecName: "console-config") pod "7ab71d8a-473b-4440-be63-d05dfe5edbac" (UID: "7ab71d8a-473b-4440-be63-d05dfe5edbac"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:49:10.353189 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.353168    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7ab71d8a-473b-4440-be63-d05dfe5edbac" (UID: "7ab71d8a-473b-4440-be63-d05dfe5edbac"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:49:10.353189 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.353175    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7ab71d8a-473b-4440-be63-d05dfe5edbac" (UID: "7ab71d8a-473b-4440-be63-d05dfe5edbac"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:49:10.353336 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.353233    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-service-ca" (OuterVolumeSpecName: "service-ca") pod "7ab71d8a-473b-4440-be63-d05dfe5edbac" (UID: "7ab71d8a-473b-4440-be63-d05dfe5edbac"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:49:10.354947 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.354920    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab71d8a-473b-4440-be63-d05dfe5edbac-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7ab71d8a-473b-4440-be63-d05dfe5edbac" (UID: "7ab71d8a-473b-4440-be63-d05dfe5edbac"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:49:10.355003 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.354984    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab71d8a-473b-4440-be63-d05dfe5edbac-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7ab71d8a-473b-4440-be63-d05dfe5edbac" (UID: "7ab71d8a-473b-4440-be63-d05dfe5edbac"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:49:10.355082 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.355067    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab71d8a-473b-4440-be63-d05dfe5edbac-kube-api-access-dfqlg" (OuterVolumeSpecName: "kube-api-access-dfqlg") pod "7ab71d8a-473b-4440-be63-d05dfe5edbac" (UID: "7ab71d8a-473b-4440-be63-d05dfe5edbac"). InnerVolumeSpecName "kube-api-access-dfqlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:49:10.453839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.453798    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ab71d8a-473b-4440-be63-d05dfe5edbac-console-serving-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:49:10.453839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.453830    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-service-ca\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:49:10.453839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.453840    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ab71d8a-473b-4440-be63-d05dfe5edbac-console-oauth-config\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:49:10.453839 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.453851    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfqlg\" (UniqueName: \"kubernetes.io/projected/7ab71d8a-473b-4440-be63-d05dfe5edbac-kube-api-access-dfqlg\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:49:10.454082 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.453860    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-oauth-serving-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:49:10.454082 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.453868    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-console-config\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:49:10.454082 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:10.453877    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ab71d8a-473b-4440-be63-d05dfe5edbac-trusted-ca-bundle\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:49:11.002071 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:11.002037    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ab71d8a-473b-4440-be63-d05dfe5edbac" containerID="361c23f386310127a43214b29b96485200e2250b78e0deeffb804760dfb5f090" exitCode=0
Jul 31 23:49:11.002495 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:11.002085    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fc4ccf46-7qdvg"
Jul 31 23:49:11.002495 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:11.002122    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fc4ccf46-7qdvg" event={"ID":"7ab71d8a-473b-4440-be63-d05dfe5edbac","Type":"ContainerDied","Data":"361c23f386310127a43214b29b96485200e2250b78e0deeffb804760dfb5f090"}
Jul 31 23:49:11.002495 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:11.002173    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fc4ccf46-7qdvg" event={"ID":"7ab71d8a-473b-4440-be63-d05dfe5edbac","Type":"ContainerDied","Data":"250792bc12d36a5d8eee23f15bd7851dc7bd0da37f201421f8c8b35dc4b9faa1"}
Jul 31 23:49:11.002495 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:11.002194    2578 scope.go:117] "RemoveContainer" containerID="361c23f386310127a43214b29b96485200e2250b78e0deeffb804760dfb5f090"
Jul 31 23:49:11.010639 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:11.010622    2578 scope.go:117] "RemoveContainer" containerID="361c23f386310127a43214b29b96485200e2250b78e0deeffb804760dfb5f090"
Jul 31 23:49:11.010893 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:49:11.010877    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361c23f386310127a43214b29b96485200e2250b78e0deeffb804760dfb5f090\": container with ID starting with 361c23f386310127a43214b29b96485200e2250b78e0deeffb804760dfb5f090 not found: ID does not exist" containerID="361c23f386310127a43214b29b96485200e2250b78e0deeffb804760dfb5f090"
Jul 31 23:49:11.010948 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:11.010902    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361c23f386310127a43214b29b96485200e2250b78e0deeffb804760dfb5f090"} err="failed to get container status \"361c23f386310127a43214b29b96485200e2250b78e0deeffb804760dfb5f090\": rpc error: code = NotFound desc = could not find container \"361c23f386310127a43214b29b96485200e2250b78e0deeffb804760dfb5f090\": container with ID starting with 361c23f386310127a43214b29b96485200e2250b78e0deeffb804760dfb5f090 not found: ID does not exist"
Jul 31 23:49:11.023180 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:11.023160    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74fc4ccf46-7qdvg"]
Jul 31 23:49:11.027240 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:11.027208    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74fc4ccf46-7qdvg"]
Jul 31 23:49:11.183886 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:11.183858    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ab71d8a-473b-4440-be63-d05dfe5edbac" path="/var/lib/kubelet/pods/7ab71d8a-473b-4440-be63-d05dfe5edbac/volumes"
Jul 31 23:49:33.376705 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.376667    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w"]
Jul 31 23:49:33.377287 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.377122    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ab71d8a-473b-4440-be63-d05dfe5edbac" containerName="console"
Jul 31 23:49:33.377287 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.377138    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ab71d8a-473b-4440-be63-d05dfe5edbac" containerName="console"
Jul 31 23:49:33.377287 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.377203    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ab71d8a-473b-4440-be63-d05dfe5edbac" containerName="console"
Jul 31 23:49:33.380682 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.380658    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w"
Jul 31 23:49:33.382684 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.382664    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 31 23:49:33.382684 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.382664    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 31 23:49:33.382924 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.382909    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2h5xk\""
Jul 31 23:49:33.387559 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.387536    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w"]
Jul 31 23:49:33.534450 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.534404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5d380b4-0d93-493e-8923-ed4c99323965-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w\" (UID: \"c5d380b4-0d93-493e-8923-ed4c99323965\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w"
Jul 31 23:49:33.534667 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.534482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4nqr\" (UniqueName: \"kubernetes.io/projected/c5d380b4-0d93-493e-8923-ed4c99323965-kube-api-access-t4nqr\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w\" (UID: \"c5d380b4-0d93-493e-8923-ed4c99323965\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w"
Jul 31 23:49:33.534667 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.534523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5d380b4-0d93-493e-8923-ed4c99323965-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w\" (UID: \"c5d380b4-0d93-493e-8923-ed4c99323965\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w"
Jul 31 23:49:33.635631 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.635532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4nqr\" (UniqueName: \"kubernetes.io/projected/c5d380b4-0d93-493e-8923-ed4c99323965-kube-api-access-t4nqr\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w\" (UID: \"c5d380b4-0d93-493e-8923-ed4c99323965\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w"
Jul 31 23:49:33.635631 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.635595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5d380b4-0d93-493e-8923-ed4c99323965-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w\" (UID: \"c5d380b4-0d93-493e-8923-ed4c99323965\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w"
Jul 31 23:49:33.635631 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.635638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5d380b4-0d93-493e-8923-ed4c99323965-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w\" (UID: \"c5d380b4-0d93-493e-8923-ed4c99323965\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w"
Jul 31 23:49:33.635994 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.635979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5d380b4-0d93-493e-8923-ed4c99323965-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w\" (UID: \"c5d380b4-0d93-493e-8923-ed4c99323965\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w"
Jul 31 23:49:33.636051 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.636034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5d380b4-0d93-493e-8923-ed4c99323965-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w\" (UID: \"c5d380b4-0d93-493e-8923-ed4c99323965\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w"
Jul 31 23:49:33.643225 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.643201    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4nqr\" (UniqueName: \"kubernetes.io/projected/c5d380b4-0d93-493e-8923-ed4c99323965-kube-api-access-t4nqr\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w\" (UID: \"c5d380b4-0d93-493e-8923-ed4c99323965\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w"
Jul 31 23:49:33.691308 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.691273    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w"
Jul 31 23:49:33.822649 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:33.822620    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w"]
Jul 31 23:49:33.825808 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:49:33.825777    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d380b4_0d93_493e_8923_ed4c99323965.slice/crio-e1d7723a13aaa639ef048bec2456ec087c93e34867fe7422ca4a0e1934e353fd WatchSource:0}: Error finding container e1d7723a13aaa639ef048bec2456ec087c93e34867fe7422ca4a0e1934e353fd: Status 404 returned error can't find the container with id e1d7723a13aaa639ef048bec2456ec087c93e34867fe7422ca4a0e1934e353fd
Jul 31 23:49:34.086541 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:34.086503    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w" event={"ID":"c5d380b4-0d93-493e-8923-ed4c99323965","Type":"ContainerStarted","Data":"e1d7723a13aaa639ef048bec2456ec087c93e34867fe7422ca4a0e1934e353fd"}
Jul 31 23:49:40.114063 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:40.113970    2578 generic.go:356] "Generic (PLEG): container finished" podID="c5d380b4-0d93-493e-8923-ed4c99323965" containerID="7b60558026438fff60cfcf126a8a79364a1dd45e313baa42bafb4040cfcc5cf5" exitCode=0
Jul 31 23:49:40.114538 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:40.114068    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w" event={"ID":"c5d380b4-0d93-493e-8923-ed4c99323965","Type":"ContainerDied","Data":"7b60558026438fff60cfcf126a8a79364a1dd45e313baa42bafb4040cfcc5cf5"}
Jul 31 23:49:42.206964 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:49:42.204423    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d380b4_0d93_493e_8923_ed4c99323965.slice/crio-09d282df450efb31e1094fbdef0e5e490415ef5ed733487388d2d080b920baad.scope\": RecentStats: unable to find data in memory cache]"
Jul 31 23:49:43.127394 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:43.127359    2578 generic.go:356] "Generic (PLEG): container finished" podID="c5d380b4-0d93-493e-8923-ed4c99323965" containerID="09d282df450efb31e1094fbdef0e5e490415ef5ed733487388d2d080b920baad" exitCode=0
Jul 31 23:49:43.127569 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:43.127453    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w" event={"ID":"c5d380b4-0d93-493e-8923-ed4c99323965","Type":"ContainerDied","Data":"09d282df450efb31e1094fbdef0e5e490415ef5ed733487388d2d080b920baad"}
Jul 31 23:49:50.164654 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:50.164619    2578 generic.go:356] "Generic (PLEG): container finished" podID="c5d380b4-0d93-493e-8923-ed4c99323965" containerID="6601706688a47327b2ae2179eee1013390bafff26690f82d48a84a4d14608d2c" exitCode=0
Jul 31 23:49:50.165141 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:50.164673    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w" event={"ID":"c5d380b4-0d93-493e-8923-ed4c99323965","Type":"ContainerDied","Data":"6601706688a47327b2ae2179eee1013390bafff26690f82d48a84a4d14608d2c"}
Jul 31 23:49:51.300114 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:51.300092    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w"
Jul 31 23:49:51.383542 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:51.383501    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4nqr\" (UniqueName: \"kubernetes.io/projected/c5d380b4-0d93-493e-8923-ed4c99323965-kube-api-access-t4nqr\") pod \"c5d380b4-0d93-493e-8923-ed4c99323965\" (UID: \"c5d380b4-0d93-493e-8923-ed4c99323965\") "
Jul 31 23:49:51.383542 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:51.383561    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5d380b4-0d93-493e-8923-ed4c99323965-bundle\") pod \"c5d380b4-0d93-493e-8923-ed4c99323965\" (UID: \"c5d380b4-0d93-493e-8923-ed4c99323965\") "
Jul 31 23:49:51.383790 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:51.383616    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5d380b4-0d93-493e-8923-ed4c99323965-util\") pod \"c5d380b4-0d93-493e-8923-ed4c99323965\" (UID: \"c5d380b4-0d93-493e-8923-ed4c99323965\") "
Jul 31 23:49:51.384275 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:51.384247    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d380b4-0d93-493e-8923-ed4c99323965-bundle" (OuterVolumeSpecName: "bundle") pod "c5d380b4-0d93-493e-8923-ed4c99323965" (UID: "c5d380b4-0d93-493e-8923-ed4c99323965"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:49:51.385857 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:51.385827    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d380b4-0d93-493e-8923-ed4c99323965-kube-api-access-t4nqr" (OuterVolumeSpecName: "kube-api-access-t4nqr") pod "c5d380b4-0d93-493e-8923-ed4c99323965" (UID: "c5d380b4-0d93-493e-8923-ed4c99323965"). InnerVolumeSpecName "kube-api-access-t4nqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:49:51.389205 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:51.389181    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d380b4-0d93-493e-8923-ed4c99323965-util" (OuterVolumeSpecName: "util") pod "c5d380b4-0d93-493e-8923-ed4c99323965" (UID: "c5d380b4-0d93-493e-8923-ed4c99323965"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:49:51.484612 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:51.484529    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5d380b4-0d93-493e-8923-ed4c99323965-util\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:49:51.484612 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:51.484563    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4nqr\" (UniqueName: \"kubernetes.io/projected/c5d380b4-0d93-493e-8923-ed4c99323965-kube-api-access-t4nqr\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:49:51.484612 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:51.484573    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5d380b4-0d93-493e-8923-ed4c99323965-bundle\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:49:52.174596 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:52.174571    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w"
Jul 31 23:49:52.174816 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:52.174596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57t47w" event={"ID":"c5d380b4-0d93-493e-8923-ed4c99323965","Type":"ContainerDied","Data":"e1d7723a13aaa639ef048bec2456ec087c93e34867fe7422ca4a0e1934e353fd"}
Jul 31 23:49:52.174816 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:52.174638    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1d7723a13aaa639ef048bec2456ec087c93e34867fe7422ca4a0e1934e353fd"
Jul 31 23:49:55.530473 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.530440    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t"]
Jul 31 23:49:55.530941 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.530751    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5d380b4-0d93-493e-8923-ed4c99323965" containerName="pull"
Jul 31 23:49:55.530941 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.530763    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5d380b4-0d93-493e-8923-ed4c99323965" containerName="pull"
Jul 31 23:49:55.530941 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.530777    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5d380b4-0d93-493e-8923-ed4c99323965" containerName="extract"
Jul 31 23:49:55.530941 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.530782    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5d380b4-0d93-493e-8923-ed4c99323965" containerName="extract"
Jul 31 23:49:55.530941 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.530791    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5d380b4-0d93-493e-8923-ed4c99323965" containerName="util"
Jul 31 23:49:55.530941 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.530811    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5d380b4-0d93-493e-8923-ed4c99323965" containerName="util"
Jul 31 23:49:55.530941 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.530861    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5d380b4-0d93-493e-8923-ed4c99323965" containerName="extract"
Jul 31 23:49:55.533677 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.533663    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t"
Jul 31 23:49:55.535628 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.535605    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-6cn49\""
Jul 31 23:49:55.535799 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.535608    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 31 23:49:55.535931 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.535915    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 31 23:49:55.537508 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.537484    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 31 23:49:55.549272 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.549241    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t"]
Jul 31 23:49:55.617035 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.617001    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/86511e6a-bb97-4f6b-a747-6490aec37dc2-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t\" (UID: \"86511e6a-bb97-4f6b-a747-6490aec37dc2\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t"
Jul 31 23:49:55.617218 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.617053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftdqs\" (UniqueName: \"kubernetes.io/projected/86511e6a-bb97-4f6b-a747-6490aec37dc2-kube-api-access-ftdqs\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t\" (UID: \"86511e6a-bb97-4f6b-a747-6490aec37dc2\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t"
Jul 31 23:49:55.718020 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.717982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/86511e6a-bb97-4f6b-a747-6490aec37dc2-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t\" (UID: \"86511e6a-bb97-4f6b-a747-6490aec37dc2\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t"
Jul 31 23:49:55.718228 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.718037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftdqs\" (UniqueName: \"kubernetes.io/projected/86511e6a-bb97-4f6b-a747-6490aec37dc2-kube-api-access-ftdqs\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t\" (UID: \"86511e6a-bb97-4f6b-a747-6490aec37dc2\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t"
Jul 31 23:49:55.720559 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.720532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/86511e6a-bb97-4f6b-a747-6490aec37dc2-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t\" (UID: \"86511e6a-bb97-4f6b-a747-6490aec37dc2\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t"
Jul 31 23:49:55.726818 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.726783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftdqs\" (UniqueName: \"kubernetes.io/projected/86511e6a-bb97-4f6b-a747-6490aec37dc2-kube-api-access-ftdqs\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t\" (UID: \"86511e6a-bb97-4f6b-a747-6490aec37dc2\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t"
Jul 31 23:49:55.843575 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.843485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t"
Jul 31 23:49:55.976418 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:55.976388    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t"]
Jul 31 23:49:55.980570 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:49:55.980545    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86511e6a_bb97_4f6b_a747_6490aec37dc2.slice/crio-02e70c8ec5398db697aaa56c3e060fff9671f1c2de3b6a6780e166822d8dd435 WatchSource:0}: Error finding container 02e70c8ec5398db697aaa56c3e060fff9671f1c2de3b6a6780e166822d8dd435: Status 404 returned error can't find the container with id 02e70c8ec5398db697aaa56c3e060fff9671f1c2de3b6a6780e166822d8dd435
Jul 31 23:49:56.189157 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:56.189126    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t" event={"ID":"86511e6a-bb97-4f6b-a747-6490aec37dc2","Type":"ContainerStarted","Data":"02e70c8ec5398db697aaa56c3e060fff9671f1c2de3b6a6780e166822d8dd435"}
Jul 31 23:49:59.202144 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.202109    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t" event={"ID":"86511e6a-bb97-4f6b-a747-6490aec37dc2","Type":"ContainerStarted","Data":"d386f559daf63d99e3fc764a61443b24bbc8daff9851c729381ab0d1358c64ca"}
Jul 31 23:49:59.202524 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.202251    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t"
Jul 31 23:49:59.219295 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.219243    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t" podStartSLOduration=1.432214433 podStartE2EDuration="4.219224819s" totalImagesPullingTime="2.787010386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:49:59.21856012 +0000 UTC m=+238.578451174" watchObservedRunningTime="2026-07-31 23:49:59.219224819 +0000 UTC m=+238.579115863"
Jul 31 23:49:59.601768 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.601669    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-csjrb"]
Jul 31 23:49:59.604979 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.604959    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-csjrb"
Jul 31 23:49:59.608287 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.608264    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-rrsdz\""
Jul 31 23:49:59.608405 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.608360    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 31 23:49:59.608912 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.608895    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 31 23:49:59.615465 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.615444    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-csjrb"]
Jul 31 23:49:59.649301 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.649264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz9w9\" (UniqueName: \"kubernetes.io/projected/2044c273-8711-4d47-8891-0ec03517bce1-kube-api-access-jz9w9\") pod \"keda-operator-77f648d694-csjrb\" (UID: \"2044c273-8711-4d47-8891-0ec03517bce1\") " pod="openshift-keda/keda-operator-77f648d694-csjrb"
Jul 31 23:49:59.649458 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.649371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2044c273-8711-4d47-8891-0ec03517bce1-cabundle0\") pod \"keda-operator-77f648d694-csjrb\" (UID: \"2044c273-8711-4d47-8891-0ec03517bce1\") " pod="openshift-keda/keda-operator-77f648d694-csjrb"
Jul 31 23:49:59.649458 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.649400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2044c273-8711-4d47-8891-0ec03517bce1-certificates\") pod \"keda-operator-77f648d694-csjrb\" (UID: \"2044c273-8711-4d47-8891-0ec03517bce1\") " pod="openshift-keda/keda-operator-77f648d694-csjrb"
Jul 31 23:49:59.749771 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.749738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2044c273-8711-4d47-8891-0ec03517bce1-cabundle0\") pod \"keda-operator-77f648d694-csjrb\" (UID: \"2044c273-8711-4d47-8891-0ec03517bce1\") " pod="openshift-keda/keda-operator-77f648d694-csjrb"
Jul 31 23:49:59.749771 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.749774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2044c273-8711-4d47-8891-0ec03517bce1-certificates\") pod \"keda-operator-77f648d694-csjrb\" (UID: \"2044c273-8711-4d47-8891-0ec03517bce1\") " pod="openshift-keda/keda-operator-77f648d694-csjrb"
Jul 31 23:49:59.750009 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.749798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz9w9\" (UniqueName: \"kubernetes.io/projected/2044c273-8711-4d47-8891-0ec03517bce1-kube-api-access-jz9w9\") pod \"keda-operator-77f648d694-csjrb\" (UID: \"2044c273-8711-4d47-8891-0ec03517bce1\") " pod="openshift-keda/keda-operator-77f648d694-csjrb"
Jul 31 23:49:59.750009 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:49:59.749926    2578 secret.go:281] references non-existent secret key: ca.crt
Jul 31 23:49:59.750009 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:49:59.749950    2578 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 31 23:49:59.750009 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:49:59.749963    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-csjrb: references non-existent secret key: ca.crt
Jul 31 23:49:59.750190 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:49:59.750076    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2044c273-8711-4d47-8891-0ec03517bce1-certificates podName:2044c273-8711-4d47-8891-0ec03517bce1 nodeName:}" failed. No retries permitted until 2026-07-31 23:50:00.250054045 +0000 UTC m=+239.609945074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/2044c273-8711-4d47-8891-0ec03517bce1-certificates") pod "keda-operator-77f648d694-csjrb" (UID: "2044c273-8711-4d47-8891-0ec03517bce1") : references non-existent secret key: ca.crt
Jul 31 23:49:59.750413 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.750394    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2044c273-8711-4d47-8891-0ec03517bce1-cabundle0\") pod \"keda-operator-77f648d694-csjrb\" (UID: \"2044c273-8711-4d47-8891-0ec03517bce1\") " pod="openshift-keda/keda-operator-77f648d694-csjrb"
Jul 31 23:49:59.758574 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.758548    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz9w9\" (UniqueName: \"kubernetes.io/projected/2044c273-8711-4d47-8891-0ec03517bce1-kube-api-access-jz9w9\") pod \"keda-operator-77f648d694-csjrb\" (UID: \"2044c273-8711-4d47-8891-0ec03517bce1\") " pod="openshift-keda/keda-operator-77f648d694-csjrb"
Jul 31 23:49:59.815444 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.815400    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-28gf2"]
Jul 31 23:49:59.819210 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.819187    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-28gf2"
Jul 31 23:49:59.820960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.820939    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 31 23:49:59.825887 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.825837    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-28gf2"]
Jul 31 23:49:59.850844 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.850815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3af2420e-424d-4e95-9d7d-f0971a8bae21-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-28gf2\" (UID: \"3af2420e-424d-4e95-9d7d-f0971a8bae21\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-28gf2"
Jul 31 23:49:59.850992 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.850866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m5pp\" (UniqueName: \"kubernetes.io/projected/3af2420e-424d-4e95-9d7d-f0971a8bae21-kube-api-access-6m5pp\") pod \"keda-metrics-apiserver-55557df8c-28gf2\" (UID: \"3af2420e-424d-4e95-9d7d-f0971a8bae21\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-28gf2"
Jul 31 23:49:59.850992 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.850951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3af2420e-424d-4e95-9d7d-f0971a8bae21-certificates\") pod \"keda-metrics-apiserver-55557df8c-28gf2\" (UID: \"3af2420e-424d-4e95-9d7d-f0971a8bae21\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-28gf2"
Jul 31 23:49:59.952110 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.952073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3af2420e-424d-4e95-9d7d-f0971a8bae21-certificates\") pod \"keda-metrics-apiserver-55557df8c-28gf2\" (UID: \"3af2420e-424d-4e95-9d7d-f0971a8bae21\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-28gf2"
Jul 31 23:49:59.952279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.952130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3af2420e-424d-4e95-9d7d-f0971a8bae21-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-28gf2\" (UID: \"3af2420e-424d-4e95-9d7d-f0971a8bae21\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-28gf2"
Jul 31 23:49:59.952279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.952185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m5pp\" (UniqueName: \"kubernetes.io/projected/3af2420e-424d-4e95-9d7d-f0971a8bae21-kube-api-access-6m5pp\") pod \"keda-metrics-apiserver-55557df8c-28gf2\" (UID: \"3af2420e-424d-4e95-9d7d-f0971a8bae21\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-28gf2"
Jul 31 23:49:59.952279 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:49:59.952238    2578 secret.go:281] references non-existent secret key: tls.crt
Jul 31 23:49:59.952279 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:49:59.952257    2578 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 31 23:49:59.952279 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:49:59.952277    2578 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 31 23:49:59.952444 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:49:59.952298    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-28gf2: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 31 23:49:59.952444 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:49:59.952363    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3af2420e-424d-4e95-9d7d-f0971a8bae21-certificates podName:3af2420e-424d-4e95-9d7d-f0971a8bae21 nodeName:}" failed. No retries permitted until 2026-07-31 23:50:00.452343796 +0000 UTC m=+239.812234817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3af2420e-424d-4e95-9d7d-f0971a8bae21-certificates") pod "keda-metrics-apiserver-55557df8c-28gf2" (UID: "3af2420e-424d-4e95-9d7d-f0971a8bae21") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 31 23:49:59.952549 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.952531    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3af2420e-424d-4e95-9d7d-f0971a8bae21-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-28gf2\" (UID: \"3af2420e-424d-4e95-9d7d-f0971a8bae21\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-28gf2"
Jul 31 23:49:59.960015 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:49:59.959989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m5pp\" (UniqueName: \"kubernetes.io/projected/3af2420e-424d-4e95-9d7d-f0971a8bae21-kube-api-access-6m5pp\") pod \"keda-metrics-apiserver-55557df8c-28gf2\" (UID: \"3af2420e-424d-4e95-9d7d-f0971a8bae21\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-28gf2"
Jul 31 23:50:00.254516 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.254431    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2044c273-8711-4d47-8891-0ec03517bce1-certificates\") pod \"keda-operator-77f648d694-csjrb\" (UID: \"2044c273-8711-4d47-8891-0ec03517bce1\") " pod="openshift-keda/keda-operator-77f648d694-csjrb"
Jul 31 23:50:00.257111 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.257087    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2044c273-8711-4d47-8891-0ec03517bce1-certificates\") pod \"keda-operator-77f648d694-csjrb\" (UID: \"2044c273-8711-4d47-8891-0ec03517bce1\") " pod="openshift-keda/keda-operator-77f648d694-csjrb"
Jul 31 23:50:00.280304 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.280272    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-tbzzk"]
Jul 31 23:50:00.283622 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.283606    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-tbzzk"
Jul 31 23:50:00.285391 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.285369    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 31 23:50:00.290442 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.290422    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-tbzzk"]
Jul 31 23:50:00.355817 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.355779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6bc4d251-5ae1-40b1-8f89-7bfccd614230-certificates\") pod \"keda-admission-56f9798bb4-tbzzk\" (UID: \"6bc4d251-5ae1-40b1-8f89-7bfccd614230\") " pod="openshift-keda/keda-admission-56f9798bb4-tbzzk"
Jul 31 23:50:00.356047 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.355854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g57vp\" (UniqueName: \"kubernetes.io/projected/6bc4d251-5ae1-40b1-8f89-7bfccd614230-kube-api-access-g57vp\") pod \"keda-admission-56f9798bb4-tbzzk\" (UID: \"6bc4d251-5ae1-40b1-8f89-7bfccd614230\") " pod="openshift-keda/keda-admission-56f9798bb4-tbzzk"
Jul 31 23:50:00.457001 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.456972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g57vp\" (UniqueName: \"kubernetes.io/projected/6bc4d251-5ae1-40b1-8f89-7bfccd614230-kube-api-access-g57vp\") pod \"keda-admission-56f9798bb4-tbzzk\" (UID: \"6bc4d251-5ae1-40b1-8f89-7bfccd614230\") " pod="openshift-keda/keda-admission-56f9798bb4-tbzzk"
Jul 31 23:50:00.457215 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.457027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3af2420e-424d-4e95-9d7d-f0971a8bae21-certificates\") pod \"keda-metrics-apiserver-55557df8c-28gf2\" (UID: \"3af2420e-424d-4e95-9d7d-f0971a8bae21\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-28gf2"
Jul 31 23:50:00.457215 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.457075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6bc4d251-5ae1-40b1-8f89-7bfccd614230-certificates\") pod \"keda-admission-56f9798bb4-tbzzk\" (UID: \"6bc4d251-5ae1-40b1-8f89-7bfccd614230\") " pod="openshift-keda/keda-admission-56f9798bb4-tbzzk"
Jul 31 23:50:00.457215 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:50:00.457197    2578 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 31 23:50:00.457215 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:50:00.457219    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-tbzzk: secret "keda-admission-webhooks-certs" not found
Jul 31 23:50:00.457402 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:50:00.457266    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bc4d251-5ae1-40b1-8f89-7bfccd614230-certificates podName:6bc4d251-5ae1-40b1-8f89-7bfccd614230 nodeName:}" failed. No retries permitted until 2026-07-31 23:50:00.957251707 +0000 UTC m=+240.317142730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/6bc4d251-5ae1-40b1-8f89-7bfccd614230-certificates") pod "keda-admission-56f9798bb4-tbzzk" (UID: "6bc4d251-5ae1-40b1-8f89-7bfccd614230") : secret "keda-admission-webhooks-certs" not found
Jul 31 23:50:00.460423 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.460383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3af2420e-424d-4e95-9d7d-f0971a8bae21-certificates\") pod \"keda-metrics-apiserver-55557df8c-28gf2\" (UID: \"3af2420e-424d-4e95-9d7d-f0971a8bae21\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-28gf2"
Jul 31 23:50:00.465803 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.465778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g57vp\" (UniqueName: \"kubernetes.io/projected/6bc4d251-5ae1-40b1-8f89-7bfccd614230-kube-api-access-g57vp\") pod \"keda-admission-56f9798bb4-tbzzk\" (UID: \"6bc4d251-5ae1-40b1-8f89-7bfccd614230\") " pod="openshift-keda/keda-admission-56f9798bb4-tbzzk"
Jul 31 23:50:00.515678 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.515553    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-csjrb"
Jul 31 23:50:00.672180 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.672148    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-csjrb"]
Jul 31 23:50:00.678596 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:50:00.678563    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2044c273_8711_4d47_8891_0ec03517bce1.slice/crio-c40273a182b8c0ba5b575347d7fcbe2d1555a90a39bba84712632f2ff010712d WatchSource:0}: Error finding container c40273a182b8c0ba5b575347d7fcbe2d1555a90a39bba84712632f2ff010712d: Status 404 returned error can't find the container with id c40273a182b8c0ba5b575347d7fcbe2d1555a90a39bba84712632f2ff010712d
Jul 31 23:50:00.732188 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.732152    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-28gf2"
Jul 31 23:50:00.886289 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.886258    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-28gf2"]
Jul 31 23:50:00.890456 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:50:00.890411    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af2420e_424d_4e95_9d7d_f0971a8bae21.slice/crio-eac266087be8fcfd652c7c5bb04c5c1a9e889346a531007a8a75eec1f9a6ca1a WatchSource:0}: Error finding container eac266087be8fcfd652c7c5bb04c5c1a9e889346a531007a8a75eec1f9a6ca1a: Status 404 returned error can't find the container with id eac266087be8fcfd652c7c5bb04c5c1a9e889346a531007a8a75eec1f9a6ca1a
Jul 31 23:50:00.964962 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.964926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6bc4d251-5ae1-40b1-8f89-7bfccd614230-certificates\") pod \"keda-admission-56f9798bb4-tbzzk\" (UID: \"6bc4d251-5ae1-40b1-8f89-7bfccd614230\") " pod="openshift-keda/keda-admission-56f9798bb4-tbzzk"
Jul 31 23:50:00.967420 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:00.967395    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6bc4d251-5ae1-40b1-8f89-7bfccd614230-certificates\") pod \"keda-admission-56f9798bb4-tbzzk\" (UID: \"6bc4d251-5ae1-40b1-8f89-7bfccd614230\") " pod="openshift-keda/keda-admission-56f9798bb4-tbzzk"
Jul 31 23:50:01.194305 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:01.194271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-tbzzk"
Jul 31 23:50:01.211423 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:01.211392    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-csjrb" event={"ID":"2044c273-8711-4d47-8891-0ec03517bce1","Type":"ContainerStarted","Data":"c40273a182b8c0ba5b575347d7fcbe2d1555a90a39bba84712632f2ff010712d"}
Jul 31 23:50:01.212621 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:01.212601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-28gf2" event={"ID":"3af2420e-424d-4e95-9d7d-f0971a8bae21","Type":"ContainerStarted","Data":"eac266087be8fcfd652c7c5bb04c5c1a9e889346a531007a8a75eec1f9a6ca1a"}
Jul 31 23:50:01.326020 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:01.325985    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-tbzzk"]
Jul 31 23:50:01.329272 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:50:01.329198    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc4d251_5ae1_40b1_8f89_7bfccd614230.slice/crio-6c3ca27c1ca77cdb6e61f9ab63936e761473a7b95d9bcefb4029a9809a9d763a WatchSource:0}: Error finding container 6c3ca27c1ca77cdb6e61f9ab63936e761473a7b95d9bcefb4029a9809a9d763a: Status 404 returned error can't find the container with id 6c3ca27c1ca77cdb6e61f9ab63936e761473a7b95d9bcefb4029a9809a9d763a
Jul 31 23:50:02.217878 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:02.217844    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-tbzzk" event={"ID":"6bc4d251-5ae1-40b1-8f89-7bfccd614230","Type":"ContainerStarted","Data":"6c3ca27c1ca77cdb6e61f9ab63936e761473a7b95d9bcefb4029a9809a9d763a"}
Jul 31 23:50:04.230154 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:04.230118    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-tbzzk" event={"ID":"6bc4d251-5ae1-40b1-8f89-7bfccd614230","Type":"ContainerStarted","Data":"2760de0ba0f967ac8423df3658a177f45e43e67e082d9bdc7d9358059acba7ff"}
Jul 31 23:50:04.230581 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:04.230194    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-tbzzk"
Jul 31 23:50:04.247883 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:04.247766    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-tbzzk" podStartSLOduration=2.281720436 podStartE2EDuration="4.247743049s" totalImagesPullingTime="1.966022613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:50:04.247239851 +0000 UTC m=+243.607130918" watchObservedRunningTime="2026-07-31 23:50:04.247743049 +0000 UTC m=+243.607634093"
Jul 31 23:50:06.240760 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:06.240709    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-csjrb" event={"ID":"2044c273-8711-4d47-8891-0ec03517bce1","Type":"ContainerStarted","Data":"86f08a76e37a9aabc8ffa034886d9f7c7a295f2600f26ba9d31967f12d173ded"}
Jul 31 23:50:06.241320 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:06.240819    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-csjrb"
Jul 31 23:50:06.242107 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:06.242087    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-28gf2" event={"ID":"3af2420e-424d-4e95-9d7d-f0971a8bae21","Type":"ContainerStarted","Data":"f7f6ec7177bebdc343f27aab7d1ac739b54ff3eff5f24d07d289208636aa5c2b"}
Jul 31 23:50:06.242225 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:06.242144    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-28gf2"
Jul 31 23:50:06.270196 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:06.270146    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-csjrb" podStartSLOduration=2.206725533 podStartE2EDuration="7.270128972s" totalImagesPullingTime="5.063403439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:50:06.254398931 +0000 UTC m=+245.614289983" watchObservedRunningTime="2026-07-31 23:50:06.270128972 +0000 UTC m=+245.630020014"
Jul 31 23:50:06.270354 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:06.270235    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-28gf2" podStartSLOduration=2.4228442980000002 podStartE2EDuration="7.270230137s" totalImagesPullingTime="4.847385839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:50:06.268797005 +0000 UTC m=+245.628688066" watchObservedRunningTime="2026-07-31 23:50:06.270230137 +0000 UTC m=+245.630121177"
Jul 31 23:50:17.250396 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:17.250369    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-28gf2"
Jul 31 23:50:20.218014 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:20.217973    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t7x7t"
Jul 31 23:50:25.237748 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:25.237651    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-tbzzk"
Jul 31 23:50:27.248394 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:27.248364    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-csjrb"
Jul 31 23:50:53.550233 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:53.550199    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m"]
Jul 31 23:50:53.553631 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:53.553614    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m"
Jul 31 23:50:53.555980 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:53.555957    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2h5xk\""
Jul 31 23:50:53.556115 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:53.555986    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 31 23:50:53.556457 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:53.556440    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 31 23:50:53.567406 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:53.567382    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m"]
Jul 31 23:50:53.595206 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:53.595175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4269a6c3-8fbd-4c3d-9993-2e4085824bec-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m\" (UID: \"4269a6c3-8fbd-4c3d-9993-2e4085824bec\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m"
Jul 31 23:50:53.595375 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:53.595216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4269a6c3-8fbd-4c3d-9993-2e4085824bec-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m\" (UID: \"4269a6c3-8fbd-4c3d-9993-2e4085824bec\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m"
Jul 31 23:50:53.595375 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:53.595235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vmnr\" (UniqueName: \"kubernetes.io/projected/4269a6c3-8fbd-4c3d-9993-2e4085824bec-kube-api-access-5vmnr\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m\" (UID: \"4269a6c3-8fbd-4c3d-9993-2e4085824bec\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m"
Jul 31 23:50:53.696587 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:53.696549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4269a6c3-8fbd-4c3d-9993-2e4085824bec-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m\" (UID: \"4269a6c3-8fbd-4c3d-9993-2e4085824bec\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m"
Jul 31 23:50:53.696587 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:53.696588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmnr\" (UniqueName: \"kubernetes.io/projected/4269a6c3-8fbd-4c3d-9993-2e4085824bec-kube-api-access-5vmnr\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m\" (UID: \"4269a6c3-8fbd-4c3d-9993-2e4085824bec\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m"
Jul 31 23:50:53.696824 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:53.696655    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4269a6c3-8fbd-4c3d-9993-2e4085824bec-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m\" (UID: \"4269a6c3-8fbd-4c3d-9993-2e4085824bec\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m"
Jul 31 23:50:53.697017 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:53.697001    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4269a6c3-8fbd-4c3d-9993-2e4085824bec-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m\" (UID: \"4269a6c3-8fbd-4c3d-9993-2e4085824bec\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m"
Jul 31 23:50:53.697077 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:53.697057    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4269a6c3-8fbd-4c3d-9993-2e4085824bec-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m\" (UID: \"4269a6c3-8fbd-4c3d-9993-2e4085824bec\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m"
Jul 31 23:50:53.706173 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:53.706149    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmnr\" (UniqueName: \"kubernetes.io/projected/4269a6c3-8fbd-4c3d-9993-2e4085824bec-kube-api-access-5vmnr\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m\" (UID: \"4269a6c3-8fbd-4c3d-9993-2e4085824bec\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m"
Jul 31 23:50:53.862672 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:53.862592    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m"
Jul 31 23:50:53.998471 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:53.998437    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m"]
Jul 31 23:50:54.001487 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:50:54.001458    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4269a6c3_8fbd_4c3d_9993_2e4085824bec.slice/crio-1c070461156df065da45f2c0b6f4a899391239ac6ed481bc88537b34b3972a58 WatchSource:0}: Error finding container 1c070461156df065da45f2c0b6f4a899391239ac6ed481bc88537b34b3972a58: Status 404 returned error can't find the container with id 1c070461156df065da45f2c0b6f4a899391239ac6ed481bc88537b34b3972a58
Jul 31 23:50:54.454531 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:54.454500    2578 generic.go:356] "Generic (PLEG): container finished" podID="4269a6c3-8fbd-4c3d-9993-2e4085824bec" containerID="4513b611f2174c99c190d285805e51c842568614f007f06f7dfb0afff7eb46df" exitCode=0
Jul 31 23:50:54.454709 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:54.454586    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m" event={"ID":"4269a6c3-8fbd-4c3d-9993-2e4085824bec","Type":"ContainerDied","Data":"4513b611f2174c99c190d285805e51c842568614f007f06f7dfb0afff7eb46df"}
Jul 31 23:50:54.454709 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:54.454637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m" event={"ID":"4269a6c3-8fbd-4c3d-9993-2e4085824bec","Type":"ContainerStarted","Data":"1c070461156df065da45f2c0b6f4a899391239ac6ed481bc88537b34b3972a58"}
Jul 31 23:50:57.475081 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:57.475042    2578 generic.go:356] "Generic (PLEG): container finished" podID="4269a6c3-8fbd-4c3d-9993-2e4085824bec" containerID="380d80ee670a3135eee31aa6ca0397a41c09c2cc1f19ac8f77a231765b6a588b" exitCode=0
Jul 31 23:50:57.475547 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:57.475124    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m" event={"ID":"4269a6c3-8fbd-4c3d-9993-2e4085824bec","Type":"ContainerDied","Data":"380d80ee670a3135eee31aa6ca0397a41c09c2cc1f19ac8f77a231765b6a588b"}
Jul 31 23:50:58.481421 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:58.481394    2578 generic.go:356] "Generic (PLEG): container finished" podID="4269a6c3-8fbd-4c3d-9993-2e4085824bec" containerID="7696869eee0de51ecbd69d22ed2a97fea67807564e8b73f6c2843278a34bb108" exitCode=0
Jul 31 23:50:58.482028 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:58.481474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m" event={"ID":"4269a6c3-8fbd-4c3d-9993-2e4085824bec","Type":"ContainerDied","Data":"7696869eee0de51ecbd69d22ed2a97fea67807564e8b73f6c2843278a34bb108"}
Jul 31 23:50:59.609741 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:59.609700    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m"
Jul 31 23:50:59.649617 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:59.649587    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vmnr\" (UniqueName: \"kubernetes.io/projected/4269a6c3-8fbd-4c3d-9993-2e4085824bec-kube-api-access-5vmnr\") pod \"4269a6c3-8fbd-4c3d-9993-2e4085824bec\" (UID: \"4269a6c3-8fbd-4c3d-9993-2e4085824bec\") "
Jul 31 23:50:59.649801 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:59.649626    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4269a6c3-8fbd-4c3d-9993-2e4085824bec-bundle\") pod \"4269a6c3-8fbd-4c3d-9993-2e4085824bec\" (UID: \"4269a6c3-8fbd-4c3d-9993-2e4085824bec\") "
Jul 31 23:50:59.649801 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:59.649674    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4269a6c3-8fbd-4c3d-9993-2e4085824bec-util\") pod \"4269a6c3-8fbd-4c3d-9993-2e4085824bec\" (UID: \"4269a6c3-8fbd-4c3d-9993-2e4085824bec\") "
Jul 31 23:50:59.650313 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:59.650283    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4269a6c3-8fbd-4c3d-9993-2e4085824bec-bundle" (OuterVolumeSpecName: "bundle") pod "4269a6c3-8fbd-4c3d-9993-2e4085824bec" (UID: "4269a6c3-8fbd-4c3d-9993-2e4085824bec"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:50:59.651853 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:59.651827    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4269a6c3-8fbd-4c3d-9993-2e4085824bec-kube-api-access-5vmnr" (OuterVolumeSpecName: "kube-api-access-5vmnr") pod "4269a6c3-8fbd-4c3d-9993-2e4085824bec" (UID: "4269a6c3-8fbd-4c3d-9993-2e4085824bec"). InnerVolumeSpecName "kube-api-access-5vmnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:50:59.655053 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:59.655024    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4269a6c3-8fbd-4c3d-9993-2e4085824bec-util" (OuterVolumeSpecName: "util") pod "4269a6c3-8fbd-4c3d-9993-2e4085824bec" (UID: "4269a6c3-8fbd-4c3d-9993-2e4085824bec"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:50:59.750970 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:59.750884    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vmnr\" (UniqueName: \"kubernetes.io/projected/4269a6c3-8fbd-4c3d-9993-2e4085824bec-kube-api-access-5vmnr\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:50:59.750970 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:59.750916    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4269a6c3-8fbd-4c3d-9993-2e4085824bec-bundle\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:50:59.750970 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:50:59.750928    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4269a6c3-8fbd-4c3d-9993-2e4085824bec-util\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:51:00.491957 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:00.491925    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m"
Jul 31 23:51:00.492148 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:00.491923    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754046l5m" event={"ID":"4269a6c3-8fbd-4c3d-9993-2e4085824bec","Type":"ContainerDied","Data":"1c070461156df065da45f2c0b6f4a899391239ac6ed481bc88537b34b3972a58"}
Jul 31 23:51:00.492148 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:00.492049    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c070461156df065da45f2c0b6f4a899391239ac6ed481bc88537b34b3972a58"
Jul 31 23:51:01.080542 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:01.080512    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Jul 31 23:51:01.085774 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:01.085750    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Jul 31 23:51:01.088037 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:01.088020    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 31 23:51:06.594404 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.594370    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mjr49"]
Jul 31 23:51:06.596778 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.594695    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4269a6c3-8fbd-4c3d-9993-2e4085824bec" containerName="util"
Jul 31 23:51:06.596778 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.594707    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4269a6c3-8fbd-4c3d-9993-2e4085824bec" containerName="util"
Jul 31 23:51:06.596778 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.594728    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4269a6c3-8fbd-4c3d-9993-2e4085824bec" containerName="pull"
Jul 31 23:51:06.596778 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.594734    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4269a6c3-8fbd-4c3d-9993-2e4085824bec" containerName="pull"
Jul 31 23:51:06.596778 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.594741    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4269a6c3-8fbd-4c3d-9993-2e4085824bec" containerName="extract"
Jul 31 23:51:06.596778 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.594747    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4269a6c3-8fbd-4c3d-9993-2e4085824bec" containerName="extract"
Jul 31 23:51:06.596778 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.594793    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4269a6c3-8fbd-4c3d-9993-2e4085824bec" containerName="extract"
Jul 31 23:51:06.597616 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.597601    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mjr49"
Jul 31 23:51:06.599898 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.599876    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-bmm8z\""
Jul 31 23:51:06.600001 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.599897    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 31 23:51:06.600001 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.599940    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:51:06.623139 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.623110    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mjr49"]
Jul 31 23:51:06.710313 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.710274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/258cf768-475a-46b3-9aa0-f5c6938fd1ed-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-mjr49\" (UID: \"258cf768-475a-46b3-9aa0-f5c6938fd1ed\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mjr49"
Jul 31 23:51:06.710473 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.710321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48fxp\" (UniqueName: \"kubernetes.io/projected/258cf768-475a-46b3-9aa0-f5c6938fd1ed-kube-api-access-48fxp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-mjr49\" (UID: \"258cf768-475a-46b3-9aa0-f5c6938fd1ed\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mjr49"
Jul 31 23:51:06.811113 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.811065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48fxp\" (UniqueName: \"kubernetes.io/projected/258cf768-475a-46b3-9aa0-f5c6938fd1ed-kube-api-access-48fxp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-mjr49\" (UID: \"258cf768-475a-46b3-9aa0-f5c6938fd1ed\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mjr49"
Jul 31 23:51:06.811298 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.811148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/258cf768-475a-46b3-9aa0-f5c6938fd1ed-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-mjr49\" (UID: \"258cf768-475a-46b3-9aa0-f5c6938fd1ed\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mjr49"
Jul 31 23:51:06.811484 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.811469    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/258cf768-475a-46b3-9aa0-f5c6938fd1ed-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-mjr49\" (UID: \"258cf768-475a-46b3-9aa0-f5c6938fd1ed\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mjr49"
Jul 31 23:51:06.819525 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.819487    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48fxp\" (UniqueName: \"kubernetes.io/projected/258cf768-475a-46b3-9aa0-f5c6938fd1ed-kube-api-access-48fxp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-mjr49\" (UID: \"258cf768-475a-46b3-9aa0-f5c6938fd1ed\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mjr49"
Jul 31 23:51:06.906691 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:06.906662    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mjr49"
Jul 31 23:51:07.059907 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:07.059615    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mjr49"]
Jul 31 23:51:07.074215 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:51:07.073627    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258cf768_475a_46b3_9aa0_f5c6938fd1ed.slice/crio-e9b260cb1e396476cf21d9bb4d22edcbcf65f6fc1cd6b968bf9a4e0daf8fb188 WatchSource:0}: Error finding container e9b260cb1e396476cf21d9bb4d22edcbcf65f6fc1cd6b968bf9a4e0daf8fb188: Status 404 returned error can't find the container with id e9b260cb1e396476cf21d9bb4d22edcbcf65f6fc1cd6b968bf9a4e0daf8fb188
Jul 31 23:51:07.077669 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:07.077628    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 23:51:07.524042 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:07.524010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mjr49" event={"ID":"258cf768-475a-46b3-9aa0-f5c6938fd1ed","Type":"ContainerStarted","Data":"e9b260cb1e396476cf21d9bb4d22edcbcf65f6fc1cd6b968bf9a4e0daf8fb188"}
Jul 31 23:51:09.536689 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:09.536650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mjr49" event={"ID":"258cf768-475a-46b3-9aa0-f5c6938fd1ed","Type":"ContainerStarted","Data":"c766826691005bc38c436c48a35b40456111a90fbf43f6a7b07e6fa7771ac034"}
Jul 31 23:51:09.557527 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:09.557472    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mjr49" podStartSLOduration=1.734516025 podStartE2EDuration="3.557450912s" totalImagesPullingTime="1.822934887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:51:09.555120458 +0000 UTC m=+308.915011500" watchObservedRunningTime="2026-07-31 23:51:09.557450912 +0000 UTC m=+308.917341956"
Jul 31 23:51:15.667028 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:15.666994    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq"]
Jul 31 23:51:15.670631 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:15.670606    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq"
Jul 31 23:51:15.672345 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:15.672322    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 31 23:51:15.672472 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:15.672410    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2h5xk\""
Jul 31 23:51:15.672665 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:15.672652    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 31 23:51:15.678360 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:15.678340    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq"]
Jul 31 23:51:15.785902 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:15.785867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a2ba67d-d2fb-43d2-8353-cdd721990e28-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq\" (UID: \"3a2ba67d-d2fb-43d2-8353-cdd721990e28\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq"
Jul 31 23:51:15.785902 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:15.785913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hckq8\" (UniqueName: \"kubernetes.io/projected/3a2ba67d-d2fb-43d2-8353-cdd721990e28-kube-api-access-hckq8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq\" (UID: \"3a2ba67d-d2fb-43d2-8353-cdd721990e28\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq"
Jul 31 23:51:15.786123 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:15.785943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a2ba67d-d2fb-43d2-8353-cdd721990e28-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq\" (UID: \"3a2ba67d-d2fb-43d2-8353-cdd721990e28\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq"
Jul 31 23:51:15.886621 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:15.886583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a2ba67d-d2fb-43d2-8353-cdd721990e28-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq\" (UID: \"3a2ba67d-d2fb-43d2-8353-cdd721990e28\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq"
Jul 31 23:51:15.886854 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:15.886635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hckq8\" (UniqueName: \"kubernetes.io/projected/3a2ba67d-d2fb-43d2-8353-cdd721990e28-kube-api-access-hckq8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq\" (UID: \"3a2ba67d-d2fb-43d2-8353-cdd721990e28\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq"
Jul 31 23:51:15.886854 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:15.886667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a2ba67d-d2fb-43d2-8353-cdd721990e28-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq\" (UID: \"3a2ba67d-d2fb-43d2-8353-cdd721990e28\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq"
Jul 31 23:51:15.887014 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:15.886993    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a2ba67d-d2fb-43d2-8353-cdd721990e28-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq\" (UID: \"3a2ba67d-d2fb-43d2-8353-cdd721990e28\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq"
Jul 31 23:51:15.887082 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:15.887059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a2ba67d-d2fb-43d2-8353-cdd721990e28-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq\" (UID: \"3a2ba67d-d2fb-43d2-8353-cdd721990e28\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq"
Jul 31 23:51:15.894667 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:15.894645    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hckq8\" (UniqueName: \"kubernetes.io/projected/3a2ba67d-d2fb-43d2-8353-cdd721990e28-kube-api-access-hckq8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq\" (UID: \"3a2ba67d-d2fb-43d2-8353-cdd721990e28\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq"
Jul 31 23:51:15.980373 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:15.980282    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq"
Jul 31 23:51:16.126670 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:16.126639    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq"]
Jul 31 23:51:16.129949 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:51:16.129907    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a2ba67d_d2fb_43d2_8353_cdd721990e28.slice/crio-609387f97cbb70b7515d4e85955d428ba45ea6c0642aca4775d30547873a1e62 WatchSource:0}: Error finding container 609387f97cbb70b7515d4e85955d428ba45ea6c0642aca4775d30547873a1e62: Status 404 returned error can't find the container with id 609387f97cbb70b7515d4e85955d428ba45ea6c0642aca4775d30547873a1e62
Jul 31 23:51:16.568290 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:16.568207    2578 generic.go:356] "Generic (PLEG): container finished" podID="3a2ba67d-d2fb-43d2-8353-cdd721990e28" containerID="6886d1a74faf4060953a8c3e23742e945c8458e549b8cdcfd3e54ad0bf3d9dd0" exitCode=0
Jul 31 23:51:16.568459 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:16.568294    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq" event={"ID":"3a2ba67d-d2fb-43d2-8353-cdd721990e28","Type":"ContainerDied","Data":"6886d1a74faf4060953a8c3e23742e945c8458e549b8cdcfd3e54ad0bf3d9dd0"}
Jul 31 23:51:16.568459 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:16.568350    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq" event={"ID":"3a2ba67d-d2fb-43d2-8353-cdd721990e28","Type":"ContainerStarted","Data":"609387f97cbb70b7515d4e85955d428ba45ea6c0642aca4775d30547873a1e62"}
Jul 31 23:51:18.580938 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:18.580899    2578 generic.go:356] "Generic (PLEG): container finished" podID="3a2ba67d-d2fb-43d2-8353-cdd721990e28" containerID="bb49a6c7dfd6fdc7bb4159b8c880224de25b5a0c1afb125c040221c0d4024c09" exitCode=0
Jul 31 23:51:18.581282 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:18.580982    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq" event={"ID":"3a2ba67d-d2fb-43d2-8353-cdd721990e28","Type":"ContainerDied","Data":"bb49a6c7dfd6fdc7bb4159b8c880224de25b5a0c1afb125c040221c0d4024c09"}
Jul 31 23:51:19.587868 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:19.587833    2578 generic.go:356] "Generic (PLEG): container finished" podID="3a2ba67d-d2fb-43d2-8353-cdd721990e28" containerID="88ca45695589712a900f15137f9900011577eede8a45c28d7d0cc5707830b397" exitCode=0
Jul 31 23:51:19.588256 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:19.587916    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq" event={"ID":"3a2ba67d-d2fb-43d2-8353-cdd721990e28","Type":"ContainerDied","Data":"88ca45695589712a900f15137f9900011577eede8a45c28d7d0cc5707830b397"}
Jul 31 23:51:20.722080 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:20.722054    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq"
Jul 31 23:51:20.829984 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:20.829951    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hckq8\" (UniqueName: \"kubernetes.io/projected/3a2ba67d-d2fb-43d2-8353-cdd721990e28-kube-api-access-hckq8\") pod \"3a2ba67d-d2fb-43d2-8353-cdd721990e28\" (UID: \"3a2ba67d-d2fb-43d2-8353-cdd721990e28\") "
Jul 31 23:51:20.830188 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:20.830025    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a2ba67d-d2fb-43d2-8353-cdd721990e28-util\") pod \"3a2ba67d-d2fb-43d2-8353-cdd721990e28\" (UID: \"3a2ba67d-d2fb-43d2-8353-cdd721990e28\") "
Jul 31 23:51:20.830188 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:20.830062    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a2ba67d-d2fb-43d2-8353-cdd721990e28-bundle\") pod \"3a2ba67d-d2fb-43d2-8353-cdd721990e28\" (UID: \"3a2ba67d-d2fb-43d2-8353-cdd721990e28\") "
Jul 31 23:51:20.830465 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:20.830433    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2ba67d-d2fb-43d2-8353-cdd721990e28-bundle" (OuterVolumeSpecName: "bundle") pod "3a2ba67d-d2fb-43d2-8353-cdd721990e28" (UID: "3a2ba67d-d2fb-43d2-8353-cdd721990e28"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:51:20.832376 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:20.832353    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a2ba67d-d2fb-43d2-8353-cdd721990e28-kube-api-access-hckq8" (OuterVolumeSpecName: "kube-api-access-hckq8") pod "3a2ba67d-d2fb-43d2-8353-cdd721990e28" (UID: "3a2ba67d-d2fb-43d2-8353-cdd721990e28"). InnerVolumeSpecName "kube-api-access-hckq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:51:20.835912 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:20.835891    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2ba67d-d2fb-43d2-8353-cdd721990e28-util" (OuterVolumeSpecName: "util") pod "3a2ba67d-d2fb-43d2-8353-cdd721990e28" (UID: "3a2ba67d-d2fb-43d2-8353-cdd721990e28"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:51:20.931385 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:20.931350    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a2ba67d-d2fb-43d2-8353-cdd721990e28-util\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:51:20.931385 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:20.931380    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a2ba67d-d2fb-43d2-8353-cdd721990e28-bundle\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:51:20.931588 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:20.931393    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hckq8\" (UniqueName: \"kubernetes.io/projected/3a2ba67d-d2fb-43d2-8353-cdd721990e28-kube-api-access-hckq8\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:51:21.599135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:21.599059    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq"
Jul 31 23:51:21.599135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:21.599074    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbhbq" event={"ID":"3a2ba67d-d2fb-43d2-8353-cdd721990e28","Type":"ContainerDied","Data":"609387f97cbb70b7515d4e85955d428ba45ea6c0642aca4775d30547873a1e62"}
Jul 31 23:51:21.599135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:21.599114    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="609387f97cbb70b7515d4e85955d428ba45ea6c0642aca4775d30547873a1e62"
Jul 31 23:51:22.223158 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.223124    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jjlrg"]
Jul 31 23:51:22.223536 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.223462    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a2ba67d-d2fb-43d2-8353-cdd721990e28" containerName="pull"
Jul 31 23:51:22.223536 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.223474    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a2ba67d-d2fb-43d2-8353-cdd721990e28" containerName="pull"
Jul 31 23:51:22.223536 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.223490    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a2ba67d-d2fb-43d2-8353-cdd721990e28" containerName="util"
Jul 31 23:51:22.223536 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.223495    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a2ba67d-d2fb-43d2-8353-cdd721990e28" containerName="util"
Jul 31 23:51:22.223536 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.223513    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a2ba67d-d2fb-43d2-8353-cdd721990e28" containerName="extract"
Jul 31 23:51:22.223536 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.223519    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a2ba67d-d2fb-43d2-8353-cdd721990e28" containerName="extract"
Jul 31 23:51:22.223746 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.223567    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a2ba67d-d2fb-43d2-8353-cdd721990e28" containerName="extract"
Jul 31 23:51:22.226294 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.226279    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jjlrg"
Jul 31 23:51:22.228067 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.228043    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 31 23:51:22.228067 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.228066    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 31 23:51:22.228394 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.228377    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-w5vkt\""
Jul 31 23:51:22.235174 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.235152    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jjlrg"]
Jul 31 23:51:22.342941 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.342902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h95l7\" (UniqueName: \"kubernetes.io/projected/8307ba46-b5ef-4547-989d-e72d96521c67-kube-api-access-h95l7\") pod \"cert-manager-55b4d954d5-jjlrg\" (UID: \"8307ba46-b5ef-4547-989d-e72d96521c67\") " pod="cert-manager/cert-manager-55b4d954d5-jjlrg"
Jul 31 23:51:22.343135 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.342957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8307ba46-b5ef-4547-989d-e72d96521c67-bound-sa-token\") pod \"cert-manager-55b4d954d5-jjlrg\" (UID: \"8307ba46-b5ef-4547-989d-e72d96521c67\") " pod="cert-manager/cert-manager-55b4d954d5-jjlrg"
Jul 31 23:51:22.443616 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.443578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h95l7\" (UniqueName: \"kubernetes.io/projected/8307ba46-b5ef-4547-989d-e72d96521c67-kube-api-access-h95l7\") pod \"cert-manager-55b4d954d5-jjlrg\" (UID: \"8307ba46-b5ef-4547-989d-e72d96521c67\") " pod="cert-manager/cert-manager-55b4d954d5-jjlrg"
Jul 31 23:51:22.443823 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.443630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8307ba46-b5ef-4547-989d-e72d96521c67-bound-sa-token\") pod \"cert-manager-55b4d954d5-jjlrg\" (UID: \"8307ba46-b5ef-4547-989d-e72d96521c67\") " pod="cert-manager/cert-manager-55b4d954d5-jjlrg"
Jul 31 23:51:22.451288 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.451257    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8307ba46-b5ef-4547-989d-e72d96521c67-bound-sa-token\") pod \"cert-manager-55b4d954d5-jjlrg\" (UID: \"8307ba46-b5ef-4547-989d-e72d96521c67\") " pod="cert-manager/cert-manager-55b4d954d5-jjlrg"
Jul 31 23:51:22.451431 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.451408    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h95l7\" (UniqueName: \"kubernetes.io/projected/8307ba46-b5ef-4547-989d-e72d96521c67-kube-api-access-h95l7\") pod \"cert-manager-55b4d954d5-jjlrg\" (UID: \"8307ba46-b5ef-4547-989d-e72d96521c67\") " pod="cert-manager/cert-manager-55b4d954d5-jjlrg"
Jul 31 23:51:22.542769 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.542653    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jjlrg"
Jul 31 23:51:22.677123 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:22.677100    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jjlrg"]
Jul 31 23:51:22.679637 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:51:22.679611    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8307ba46_b5ef_4547_989d_e72d96521c67.slice/crio-5caf6a21a01d89db76664ce07baada17076745a98242876ae55fefbf5d0259a4 WatchSource:0}: Error finding container 5caf6a21a01d89db76664ce07baada17076745a98242876ae55fefbf5d0259a4: Status 404 returned error can't find the container with id 5caf6a21a01d89db76664ce07baada17076745a98242876ae55fefbf5d0259a4
Jul 31 23:51:23.612638 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:23.612599    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jjlrg" event={"ID":"8307ba46-b5ef-4547-989d-e72d96521c67","Type":"ContainerStarted","Data":"5caf6a21a01d89db76664ce07baada17076745a98242876ae55fefbf5d0259a4"}
Jul 31 23:51:25.623282 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:25.623248    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jjlrg" event={"ID":"8307ba46-b5ef-4547-989d-e72d96521c67","Type":"ContainerStarted","Data":"82cc245afe45bf89271b0df64fc6cf4a06f12befb13022008d445d13b2651f2f"}
Jul 31 23:51:25.639376 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:25.639324    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-jjlrg" podStartSLOduration=1.18028477 podStartE2EDuration="3.639306683s" totalImagesPullingTime="2.459021913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:51:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:51:25.638663814 +0000 UTC m=+324.998554854" watchObservedRunningTime="2026-07-31 23:51:25.639306683 +0000 UTC m=+324.999197724"
Jul 31 23:51:42.710285 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.710248    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"]
Jul 31 23:51:42.718443 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.718417    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.721410 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.721390    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-q6dn7\""
Jul 31 23:51:42.721702 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.721685    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 31 23:51:42.721843 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.721829    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 31 23:51:42.721976 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.721962    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 31 23:51:42.722015 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.721968    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 31 23:51:42.722227 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.722204    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 31 23:51:42.722328 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.722313    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 31 23:51:42.743133 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.743105    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"]
Jul 31 23:51:42.812195 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.812159    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d964a45-cbe1-4201-a51f-797deac2acf4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.812399 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.812206    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9d964a45-cbe1-4201-a51f-797deac2acf4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.812399 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.812279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.812399 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.812337    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.812520 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.812402    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.812520 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.812457    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.812520 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.812483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndztb\" (UniqueName: \"kubernetes.io/projected/9d964a45-cbe1-4201-a51f-797deac2acf4-kube-api-access-ndztb\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.913195 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.913161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.913369 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.913206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.913369 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.913226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndztb\" (UniqueName: \"kubernetes.io/projected/9d964a45-cbe1-4201-a51f-797deac2acf4-kube-api-access-ndztb\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.913369 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.913253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d964a45-cbe1-4201-a51f-797deac2acf4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.913369 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.913277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9d964a45-cbe1-4201-a51f-797deac2acf4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.913369 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.913307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.913369 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.913333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.914007 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.913981    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.916290 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.916267    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d964a45-cbe1-4201-a51f-797deac2acf4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.916385 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.916348    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.916467 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.916452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.916509 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.916456    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9d964a45-cbe1-4201-a51f-797deac2acf4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.920708 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.920687    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:42.921000 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:42.920980    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndztb\" (UniqueName: \"kubernetes.io/projected/9d964a45-cbe1-4201-a51f-797deac2acf4-kube-api-access-ndztb\") pod \"istiod-openshift-gateway-7b7995d5cd-lz85g\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:43.027923 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:43.027827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:43.188553 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:43.188529    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"]
Jul 31 23:51:43.191113 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:51:43.191089    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d964a45_cbe1_4201_a51f_797deac2acf4.slice/crio-af6f0c9c9534edf6fc1d7d20c0ff759b65b7e46d0518cb5df41a9f23de7cbce6 WatchSource:0}: Error finding container af6f0c9c9534edf6fc1d7d20c0ff759b65b7e46d0518cb5df41a9f23de7cbce6: Status 404 returned error can't find the container with id af6f0c9c9534edf6fc1d7d20c0ff759b65b7e46d0518cb5df41a9f23de7cbce6
Jul 31 23:51:43.704519 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:43.704487    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g" event={"ID":"9d964a45-cbe1-4201-a51f-797deac2acf4","Type":"ContainerStarted","Data":"af6f0c9c9534edf6fc1d7d20c0ff759b65b7e46d0518cb5df41a9f23de7cbce6"}
Jul 31 23:51:46.045825 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:46.045789    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 31 23:51:46.046199 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:46.045850    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 31 23:51:46.723884 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:46.723843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g" event={"ID":"9d964a45-cbe1-4201-a51f-797deac2acf4","Type":"ContainerStarted","Data":"4859851566cff37c3ee2337f81c4b768fa513fb3d17024a666c1e8472458abda"}
Jul 31 23:51:46.724274 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:46.724251    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:51:46.747788 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:46.747732    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g" podStartSLOduration=1.895431372 podStartE2EDuration="4.747702355s" totalImagesPullingTime="2.852270983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:51:46.744883435 +0000 UTC m=+346.104774499" watchObservedRunningTime="2026-07-31 23:51:46.747702355 +0000 UTC m=+346.107593396"
Jul 31 23:51:47.730428 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:51:47.730397    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:52:01.070061 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.070027    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8"]
Jul 31 23:52:01.074646 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.074624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8"
Jul 31 23:52:01.076602 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.076580    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 31 23:52:01.076764 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.076583    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2h5xk\""
Jul 31 23:52:01.076764 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.076583    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 31 23:52:01.082687 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.082661    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8"]
Jul 31 23:52:01.164502 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.164475    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv"]
Jul 31 23:52:01.166191 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.166164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cbd7da5-9146-454c-946b-47caae76240f-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8\" (UID: \"4cbd7da5-9146-454c-946b-47caae76240f\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8"
Jul 31 23:52:01.166315 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.166204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk7kg\" (UniqueName: \"kubernetes.io/projected/4cbd7da5-9146-454c-946b-47caae76240f-kube-api-access-pk7kg\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8\" (UID: \"4cbd7da5-9146-454c-946b-47caae76240f\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8"
Jul 31 23:52:01.166315 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.166293    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cbd7da5-9146-454c-946b-47caae76240f-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8\" (UID: \"4cbd7da5-9146-454c-946b-47caae76240f\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8"
Jul 31 23:52:01.167592 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.167578    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv"
Jul 31 23:52:01.176228 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.176203    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv"]
Jul 31 23:52:01.264247 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.264215    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6"]
Jul 31 23:52:01.267348 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.267331    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6"
Jul 31 23:52:01.267460 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.267349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48c3c95b-bbff-44f6-9d43-cb124ccaa9c8-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv\" (UID: \"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv"
Jul 31 23:52:01.267460 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.267385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cbd7da5-9146-454c-946b-47caae76240f-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8\" (UID: \"4cbd7da5-9146-454c-946b-47caae76240f\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8"
Jul 31 23:52:01.267460 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.267403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48c3c95b-bbff-44f6-9d43-cb124ccaa9c8-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv\" (UID: \"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv"
Jul 31 23:52:01.267566 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.267494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b8zs\" (UniqueName: \"kubernetes.io/projected/48c3c95b-bbff-44f6-9d43-cb124ccaa9c8-kube-api-access-6b8zs\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv\" (UID: \"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv"
Jul 31 23:52:01.267621 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.267603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cbd7da5-9146-454c-946b-47caae76240f-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8\" (UID: \"4cbd7da5-9146-454c-946b-47caae76240f\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8"
Jul 31 23:52:01.267658 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.267638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk7kg\" (UniqueName: \"kubernetes.io/projected/4cbd7da5-9146-454c-946b-47caae76240f-kube-api-access-pk7kg\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8\" (UID: \"4cbd7da5-9146-454c-946b-47caae76240f\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8"
Jul 31 23:52:01.267827 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.267810    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cbd7da5-9146-454c-946b-47caae76240f-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8\" (UID: \"4cbd7da5-9146-454c-946b-47caae76240f\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8"
Jul 31 23:52:01.267959 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.267938    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cbd7da5-9146-454c-946b-47caae76240f-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8\" (UID: \"4cbd7da5-9146-454c-946b-47caae76240f\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8"
Jul 31 23:52:01.276531 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.276503    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6"]
Jul 31 23:52:01.278325 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.278301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk7kg\" (UniqueName: \"kubernetes.io/projected/4cbd7da5-9146-454c-946b-47caae76240f-kube-api-access-pk7kg\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8\" (UID: \"4cbd7da5-9146-454c-946b-47caae76240f\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8"
Jul 31 23:52:01.368319 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.368241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b8zs\" (UniqueName: \"kubernetes.io/projected/48c3c95b-bbff-44f6-9d43-cb124ccaa9c8-kube-api-access-6b8zs\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv\" (UID: \"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv"
Jul 31 23:52:01.368319 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.368276    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc"]
Jul 31 23:52:01.368319 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.368305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/943bb77e-5117-4b72-af58-76cbb352ac52-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6\" (UID: \"943bb77e-5117-4b72-af58-76cbb352ac52\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6"
Jul 31 23:52:01.368596 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.368338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48c3c95b-bbff-44f6-9d43-cb124ccaa9c8-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv\" (UID: \"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv"
Jul 31 23:52:01.368596 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.368357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48c3c95b-bbff-44f6-9d43-cb124ccaa9c8-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv\" (UID: \"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv"
Jul 31 23:52:01.368596 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.368374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr2q9\" (UniqueName: \"kubernetes.io/projected/943bb77e-5117-4b72-af58-76cbb352ac52-kube-api-access-dr2q9\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6\" (UID: \"943bb77e-5117-4b72-af58-76cbb352ac52\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6"
Jul 31 23:52:01.368596 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.368393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/943bb77e-5117-4b72-af58-76cbb352ac52-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6\" (UID: \"943bb77e-5117-4b72-af58-76cbb352ac52\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6"
Jul 31 23:52:01.368848 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.368795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48c3c95b-bbff-44f6-9d43-cb124ccaa9c8-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv\" (UID: \"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv"
Jul 31 23:52:01.368848 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.368805    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48c3c95b-bbff-44f6-9d43-cb124ccaa9c8-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv\" (UID: \"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv"
Jul 31 23:52:01.371649 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.371632    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc"
Jul 31 23:52:01.375334 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.375311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b8zs\" (UniqueName: \"kubernetes.io/projected/48c3c95b-bbff-44f6-9d43-cb124ccaa9c8-kube-api-access-6b8zs\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv\" (UID: \"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv"
Jul 31 23:52:01.379354 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.379328    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc"]
Jul 31 23:52:01.385788 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.385760    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8"
Jul 31 23:52:01.469631 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.469603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr2q9\" (UniqueName: \"kubernetes.io/projected/943bb77e-5117-4b72-af58-76cbb352ac52-kube-api-access-dr2q9\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6\" (UID: \"943bb77e-5117-4b72-af58-76cbb352ac52\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6"
Jul 31 23:52:01.469631 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.469634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/943bb77e-5117-4b72-af58-76cbb352ac52-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6\" (UID: \"943bb77e-5117-4b72-af58-76cbb352ac52\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6"
Jul 31 23:52:01.469891 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.469661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/015b3f73-2132-4d6f-b69e-0a77168b111d-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc\" (UID: \"015b3f73-2132-4d6f-b69e-0a77168b111d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc"
Jul 31 23:52:01.469891 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.469743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85chd\" (UniqueName: \"kubernetes.io/projected/015b3f73-2132-4d6f-b69e-0a77168b111d-kube-api-access-85chd\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc\" (UID: \"015b3f73-2132-4d6f-b69e-0a77168b111d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc"
Jul 31 23:52:01.469891 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.469782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/943bb77e-5117-4b72-af58-76cbb352ac52-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6\" (UID: \"943bb77e-5117-4b72-af58-76cbb352ac52\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6"
Jul 31 23:52:01.469891 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.469814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/015b3f73-2132-4d6f-b69e-0a77168b111d-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc\" (UID: \"015b3f73-2132-4d6f-b69e-0a77168b111d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc"
Jul 31 23:52:01.470134 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.469970    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/943bb77e-5117-4b72-af58-76cbb352ac52-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6\" (UID: \"943bb77e-5117-4b72-af58-76cbb352ac52\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6"
Jul 31 23:52:01.470134 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.470073    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/943bb77e-5117-4b72-af58-76cbb352ac52-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6\" (UID: \"943bb77e-5117-4b72-af58-76cbb352ac52\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6"
Jul 31 23:52:01.478057 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.478030    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv"
Jul 31 23:52:01.481241 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.481196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr2q9\" (UniqueName: \"kubernetes.io/projected/943bb77e-5117-4b72-af58-76cbb352ac52-kube-api-access-dr2q9\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6\" (UID: \"943bb77e-5117-4b72-af58-76cbb352ac52\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6"
Jul 31 23:52:01.527227 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.527201    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8"]
Jul 31 23:52:01.531212 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:52:01.530176    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbd7da5_9146_454c_946b_47caae76240f.slice/crio-a91d87d411b9616e2e105434a88e6419359f94b6b7250ce40be6ae947d61380d WatchSource:0}: Error finding container a91d87d411b9616e2e105434a88e6419359f94b6b7250ce40be6ae947d61380d: Status 404 returned error can't find the container with id a91d87d411b9616e2e105434a88e6419359f94b6b7250ce40be6ae947d61380d
Jul 31 23:52:01.570199 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.570174    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/015b3f73-2132-4d6f-b69e-0a77168b111d-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc\" (UID: \"015b3f73-2132-4d6f-b69e-0a77168b111d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc"
Jul 31 23:52:01.570326 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.570262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85chd\" (UniqueName: \"kubernetes.io/projected/015b3f73-2132-4d6f-b69e-0a77168b111d-kube-api-access-85chd\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc\" (UID: \"015b3f73-2132-4d6f-b69e-0a77168b111d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc"
Jul 31 23:52:01.570326 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.570310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/015b3f73-2132-4d6f-b69e-0a77168b111d-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc\" (UID: \"015b3f73-2132-4d6f-b69e-0a77168b111d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc"
Jul 31 23:52:01.570818 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.570560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/015b3f73-2132-4d6f-b69e-0a77168b111d-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc\" (UID: \"015b3f73-2132-4d6f-b69e-0a77168b111d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc"
Jul 31 23:52:01.570818 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.570587    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/015b3f73-2132-4d6f-b69e-0a77168b111d-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc\" (UID: \"015b3f73-2132-4d6f-b69e-0a77168b111d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc"
Jul 31 23:52:01.576871 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.576851    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6"
Jul 31 23:52:01.578781 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.578758    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85chd\" (UniqueName: \"kubernetes.io/projected/015b3f73-2132-4d6f-b69e-0a77168b111d-kube-api-access-85chd\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc\" (UID: \"015b3f73-2132-4d6f-b69e-0a77168b111d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc"
Jul 31 23:52:01.617697 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.617637    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv"]
Jul 31 23:52:01.621355 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:52:01.621282    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c3c95b_bbff_44f6_9d43_cb124ccaa9c8.slice/crio-c8f8461d2cec137d9d50fe3e71c1790d0a1606da71cda814f62e7f723ee408dc WatchSource:0}: Error finding container c8f8461d2cec137d9d50fe3e71c1790d0a1606da71cda814f62e7f723ee408dc: Status 404 returned error can't find the container with id c8f8461d2cec137d9d50fe3e71c1790d0a1606da71cda814f62e7f723ee408dc
Jul 31 23:52:01.689678 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.689650    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc"
Jul 31 23:52:01.749321 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.749278    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6"]
Jul 31 23:52:01.754615 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:52:01.754582    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943bb77e_5117_4b72_af58_76cbb352ac52.slice/crio-4510ac3fc3e3455d873f08304ddf9218ea3c8c9a53d76864d66f45e7beb5947c WatchSource:0}: Error finding container 4510ac3fc3e3455d873f08304ddf9218ea3c8c9a53d76864d66f45e7beb5947c: Status 404 returned error can't find the container with id 4510ac3fc3e3455d873f08304ddf9218ea3c8c9a53d76864d66f45e7beb5947c
Jul 31 23:52:01.795443 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.795117    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv" event={"ID":"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8","Type":"ContainerStarted","Data":"6066e6a0941a5602ae932d61be2afe6a9e40e532abf07d350f9525cee685daa6"}
Jul 31 23:52:01.795443 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.795198    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv" event={"ID":"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8","Type":"ContainerStarted","Data":"c8f8461d2cec137d9d50fe3e71c1790d0a1606da71cda814f62e7f723ee408dc"}
Jul 31 23:52:01.797224 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.797197    2578 generic.go:356] "Generic (PLEG): container finished" podID="4cbd7da5-9146-454c-946b-47caae76240f" containerID="8360974800fff2594f90439ba2c6c6ca3075952e76ae2e13943a81d0cb299bba" exitCode=0
Jul 31 23:52:01.797373 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.797235    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8" event={"ID":"4cbd7da5-9146-454c-946b-47caae76240f","Type":"ContainerDied","Data":"8360974800fff2594f90439ba2c6c6ca3075952e76ae2e13943a81d0cb299bba"}
Jul 31 23:52:01.797373 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.797283    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8" event={"ID":"4cbd7da5-9146-454c-946b-47caae76240f","Type":"ContainerStarted","Data":"a91d87d411b9616e2e105434a88e6419359f94b6b7250ce40be6ae947d61380d"}
Jul 31 23:52:01.799279 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.799244    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6" event={"ID":"943bb77e-5117-4b72-af58-76cbb352ac52","Type":"ContainerStarted","Data":"4510ac3fc3e3455d873f08304ddf9218ea3c8c9a53d76864d66f45e7beb5947c"}
Jul 31 23:52:01.857591 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:01.857566    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc"]
Jul 31 23:52:01.866860 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:52:01.866833    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015b3f73_2132_4d6f_b69e_0a77168b111d.slice/crio-574f0adb1fa1a6349764fc9b688cef131f7759b88ebd56a34afda3f96e03cd87 WatchSource:0}: Error finding container 574f0adb1fa1a6349764fc9b688cef131f7759b88ebd56a34afda3f96e03cd87: Status 404 returned error can't find the container with id 574f0adb1fa1a6349764fc9b688cef131f7759b88ebd56a34afda3f96e03cd87
Jul 31 23:52:02.806009 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:02.805971    2578 generic.go:356] "Generic (PLEG): container finished" podID="943bb77e-5117-4b72-af58-76cbb352ac52" containerID="bf1c7f473d2aa07217dc1d1948fb4bfc281456491c348b874f5b15d63c651879" exitCode=0
Jul 31 23:52:02.806455 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:02.806052    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6" event={"ID":"943bb77e-5117-4b72-af58-76cbb352ac52","Type":"ContainerDied","Data":"bf1c7f473d2aa07217dc1d1948fb4bfc281456491c348b874f5b15d63c651879"}
Jul 31 23:52:02.807670 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:02.807629    2578 generic.go:356] "Generic (PLEG): container finished" podID="48c3c95b-bbff-44f6-9d43-cb124ccaa9c8" containerID="6066e6a0941a5602ae932d61be2afe6a9e40e532abf07d350f9525cee685daa6" exitCode=0
Jul 31 23:52:02.807753 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:02.807689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv" event={"ID":"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8","Type":"ContainerDied","Data":"6066e6a0941a5602ae932d61be2afe6a9e40e532abf07d350f9525cee685daa6"}
Jul 31 23:52:02.809352 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:02.809313    2578 generic.go:356] "Generic (PLEG): container finished" podID="015b3f73-2132-4d6f-b69e-0a77168b111d" containerID="39846ba8ed5d04d0c5f079a8686512e4c7d2b6552db377382b517aac24426917" exitCode=0
Jul 31 23:52:02.809418 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:02.809349    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc" event={"ID":"015b3f73-2132-4d6f-b69e-0a77168b111d","Type":"ContainerDied","Data":"39846ba8ed5d04d0c5f079a8686512e4c7d2b6552db377382b517aac24426917"}
Jul 31 23:52:02.809418 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:02.809383    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc" event={"ID":"015b3f73-2132-4d6f-b69e-0a77168b111d","Type":"ContainerStarted","Data":"574f0adb1fa1a6349764fc9b688cef131f7759b88ebd56a34afda3f96e03cd87"}
Jul 31 23:52:03.817372 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:03.817333    2578 generic.go:356] "Generic (PLEG): container finished" podID="4cbd7da5-9146-454c-946b-47caae76240f" containerID="abedafad60487bacabcc6b224da5c30ca969b17334edbdb9b7b49af0d42f2f6b" exitCode=0
Jul 31 23:52:03.817811 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:03.817414    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8" event={"ID":"4cbd7da5-9146-454c-946b-47caae76240f","Type":"ContainerDied","Data":"abedafad60487bacabcc6b224da5c30ca969b17334edbdb9b7b49af0d42f2f6b"}
Jul 31 23:52:03.821150 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:03.821104    2578 generic.go:356] "Generic (PLEG): container finished" podID="943bb77e-5117-4b72-af58-76cbb352ac52" containerID="9c48bcaa243d6045ecf077854117954baaf6d946ff4942a7b5987d0338f50385" exitCode=0
Jul 31 23:52:03.821150 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:03.821144    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6" event={"ID":"943bb77e-5117-4b72-af58-76cbb352ac52","Type":"ContainerDied","Data":"9c48bcaa243d6045ecf077854117954baaf6d946ff4942a7b5987d0338f50385"}
Jul 31 23:52:04.829729 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:04.829684    2578 generic.go:356] "Generic (PLEG): container finished" podID="943bb77e-5117-4b72-af58-76cbb352ac52" containerID="1d5602b35ef623c0ea2c6d0f7f0a75d5f691cd29f67b15c517a7f9881e3e176e" exitCode=0
Jul 31 23:52:04.830113 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:04.829836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6" event={"ID":"943bb77e-5117-4b72-af58-76cbb352ac52","Type":"ContainerDied","Data":"1d5602b35ef623c0ea2c6d0f7f0a75d5f691cd29f67b15c517a7f9881e3e176e"}
Jul 31 23:52:04.831473 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:04.831452    2578 generic.go:356] "Generic (PLEG): container finished" podID="48c3c95b-bbff-44f6-9d43-cb124ccaa9c8" containerID="12a4d6b95fd193176d422fba89240fc78420f4aa7682cf46292557ab5f98af98" exitCode=0
Jul 31 23:52:04.831594 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:04.831528    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv" event={"ID":"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8","Type":"ContainerDied","Data":"12a4d6b95fd193176d422fba89240fc78420f4aa7682cf46292557ab5f98af98"}
Jul 31 23:52:04.833635 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:04.833618    2578 generic.go:356] "Generic (PLEG): container finished" podID="4cbd7da5-9146-454c-946b-47caae76240f" containerID="8f6c683ceaf290a14b2e5356c4367feccadcd8e44fc7d57d66d0421170e9a0d8" exitCode=0
Jul 31 23:52:04.833707 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:04.833681    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8" event={"ID":"4cbd7da5-9146-454c-946b-47caae76240f","Type":"ContainerDied","Data":"8f6c683ceaf290a14b2e5356c4367feccadcd8e44fc7d57d66d0421170e9a0d8"}
Jul 31 23:52:04.835395 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:04.835377    2578 generic.go:356] "Generic (PLEG): container finished" podID="015b3f73-2132-4d6f-b69e-0a77168b111d" containerID="242c1a48a1537d30ce680a579899bf54400c2f8994470c52021bb9b208cb8990" exitCode=0
Jul 31 23:52:04.835479 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:04.835409    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc" event={"ID":"015b3f73-2132-4d6f-b69e-0a77168b111d","Type":"ContainerDied","Data":"242c1a48a1537d30ce680a579899bf54400c2f8994470c52021bb9b208cb8990"}
Jul 31 23:52:05.843029 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:05.842997    2578 generic.go:356] "Generic (PLEG): container finished" podID="48c3c95b-bbff-44f6-9d43-cb124ccaa9c8" containerID="54fc99672e4b67a04c50393354f2e9caf66aab1132ade52f50a4c7fb4821a418" exitCode=0
Jul 31 23:52:05.843488 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:05.843088    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv" event={"ID":"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8","Type":"ContainerDied","Data":"54fc99672e4b67a04c50393354f2e9caf66aab1132ade52f50a4c7fb4821a418"}
Jul 31 23:52:05.845270 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:05.845248    2578 generic.go:356] "Generic (PLEG): container finished" podID="015b3f73-2132-4d6f-b69e-0a77168b111d" containerID="86e5e7b14e2d8927d2de5887253947d67a27b217e841e4adc7424d293db0e08d" exitCode=0
Jul 31 23:52:05.845399 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:05.845330    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc" event={"ID":"015b3f73-2132-4d6f-b69e-0a77168b111d","Type":"ContainerDied","Data":"86e5e7b14e2d8927d2de5887253947d67a27b217e841e4adc7424d293db0e08d"}
Jul 31 23:52:05.889761 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:05.889705    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c56c5646b-pkdsf"]
Jul 31 23:52:05.893203 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:05.893172    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:05.904967 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:05.904903    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c56c5646b-pkdsf"]
Jul 31 23:52:05.988468 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:05.988445    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8"
Jul 31 23:52:06.012659 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.012636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/361ef423-1058-4296-81de-f7df57bee743-service-ca\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.012822 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.012668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/361ef423-1058-4296-81de-f7df57bee743-console-oauth-config\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.012822 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.012697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361ef423-1058-4296-81de-f7df57bee743-trusted-ca-bundle\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.012822 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.012751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/361ef423-1058-4296-81de-f7df57bee743-oauth-serving-cert\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.012822 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.012807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/361ef423-1058-4296-81de-f7df57bee743-console-config\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.013010 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.012846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd9rg\" (UniqueName: \"kubernetes.io/projected/361ef423-1058-4296-81de-f7df57bee743-kube-api-access-kd9rg\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.013010 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.012904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/361ef423-1058-4296-81de-f7df57bee743-console-serving-cert\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.016112 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.016094    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6"
Jul 31 23:52:06.113382 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.113300    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cbd7da5-9146-454c-946b-47caae76240f-bundle\") pod \"4cbd7da5-9146-454c-946b-47caae76240f\" (UID: \"4cbd7da5-9146-454c-946b-47caae76240f\") "
Jul 31 23:52:06.113382 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.113358    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cbd7da5-9146-454c-946b-47caae76240f-util\") pod \"4cbd7da5-9146-454c-946b-47caae76240f\" (UID: \"4cbd7da5-9146-454c-946b-47caae76240f\") "
Jul 31 23:52:06.113598 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.113428    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pk7kg\" (UniqueName: \"kubernetes.io/projected/4cbd7da5-9146-454c-946b-47caae76240f-kube-api-access-pk7kg\") pod \"4cbd7da5-9146-454c-946b-47caae76240f\" (UID: \"4cbd7da5-9146-454c-946b-47caae76240f\") "
Jul 31 23:52:06.113598 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.113461    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/943bb77e-5117-4b72-af58-76cbb352ac52-util\") pod \"943bb77e-5117-4b72-af58-76cbb352ac52\" (UID: \"943bb77e-5117-4b72-af58-76cbb352ac52\") "
Jul 31 23:52:06.113598 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.113486    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/943bb77e-5117-4b72-af58-76cbb352ac52-bundle\") pod \"943bb77e-5117-4b72-af58-76cbb352ac52\" (UID: \"943bb77e-5117-4b72-af58-76cbb352ac52\") "
Jul 31 23:52:06.113598 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.113526    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr2q9\" (UniqueName: \"kubernetes.io/projected/943bb77e-5117-4b72-af58-76cbb352ac52-kube-api-access-dr2q9\") pod \"943bb77e-5117-4b72-af58-76cbb352ac52\" (UID: \"943bb77e-5117-4b72-af58-76cbb352ac52\") "
Jul 31 23:52:06.113813 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.113638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/361ef423-1058-4296-81de-f7df57bee743-service-ca\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.113813 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.113679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/361ef423-1058-4296-81de-f7df57bee743-console-oauth-config\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.113813 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.113733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361ef423-1058-4296-81de-f7df57bee743-trusted-ca-bundle\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.113950 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.113864    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbd7da5-9146-454c-946b-47caae76240f-bundle" (OuterVolumeSpecName: "bundle") pod "4cbd7da5-9146-454c-946b-47caae76240f" (UID: "4cbd7da5-9146-454c-946b-47caae76240f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:52:06.113950 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.113918    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/361ef423-1058-4296-81de-f7df57bee743-oauth-serving-cert\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.114042 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.114024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/361ef423-1058-4296-81de-f7df57bee743-console-config\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.114095 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.114074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd9rg\" (UniqueName: \"kubernetes.io/projected/361ef423-1058-4296-81de-f7df57bee743-kube-api-access-kd9rg\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.114146 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.114134    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/361ef423-1058-4296-81de-f7df57bee743-console-serving-cert\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.114201 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.114190    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cbd7da5-9146-454c-946b-47caae76240f-bundle\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:06.114760 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.114494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/361ef423-1058-4296-81de-f7df57bee743-service-ca\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.114874 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.114759    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/361ef423-1058-4296-81de-f7df57bee743-oauth-serving-cert\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.114874 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.114802    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361ef423-1058-4296-81de-f7df57bee743-trusted-ca-bundle\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.115126 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.115074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/361ef423-1058-4296-81de-f7df57bee743-console-config\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.115126 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.115076    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943bb77e-5117-4b72-af58-76cbb352ac52-bundle" (OuterVolumeSpecName: "bundle") pod "943bb77e-5117-4b72-af58-76cbb352ac52" (UID: "943bb77e-5117-4b72-af58-76cbb352ac52"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:52:06.117209 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.117182    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbd7da5-9146-454c-946b-47caae76240f-kube-api-access-pk7kg" (OuterVolumeSpecName: "kube-api-access-pk7kg") pod "4cbd7da5-9146-454c-946b-47caae76240f" (UID: "4cbd7da5-9146-454c-946b-47caae76240f"). InnerVolumeSpecName "kube-api-access-pk7kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:52:06.117410 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.117387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/361ef423-1058-4296-81de-f7df57bee743-console-serving-cert\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.117506 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.117386    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943bb77e-5117-4b72-af58-76cbb352ac52-kube-api-access-dr2q9" (OuterVolumeSpecName: "kube-api-access-dr2q9") pod "943bb77e-5117-4b72-af58-76cbb352ac52" (UID: "943bb77e-5117-4b72-af58-76cbb352ac52"). InnerVolumeSpecName "kube-api-access-dr2q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:52:06.117669 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.117651    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/361ef423-1058-4296-81de-f7df57bee743-console-oauth-config\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.120437 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.120415    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbd7da5-9146-454c-946b-47caae76240f-util" (OuterVolumeSpecName: "util") pod "4cbd7da5-9146-454c-946b-47caae76240f" (UID: "4cbd7da5-9146-454c-946b-47caae76240f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:52:06.122426 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.122402    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943bb77e-5117-4b72-af58-76cbb352ac52-util" (OuterVolumeSpecName: "util") pod "943bb77e-5117-4b72-af58-76cbb352ac52" (UID: "943bb77e-5117-4b72-af58-76cbb352ac52"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:52:06.136896 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.136869    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd9rg\" (UniqueName: \"kubernetes.io/projected/361ef423-1058-4296-81de-f7df57bee743-kube-api-access-kd9rg\") pod \"console-7c56c5646b-pkdsf\" (UID: \"361ef423-1058-4296-81de-f7df57bee743\") " pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.204744 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.204693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:06.215443 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.215420    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cbd7da5-9146-454c-946b-47caae76240f-util\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:06.215508 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.215449    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk7kg\" (UniqueName: \"kubernetes.io/projected/4cbd7da5-9146-454c-946b-47caae76240f-kube-api-access-pk7kg\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:06.215508 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.215461    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/943bb77e-5117-4b72-af58-76cbb352ac52-util\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:06.215508 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.215471    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/943bb77e-5117-4b72-af58-76cbb352ac52-bundle\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:06.215508 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.215479    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr2q9\" (UniqueName: \"kubernetes.io/projected/943bb77e-5117-4b72-af58-76cbb352ac52-kube-api-access-dr2q9\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:06.348511 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.343000    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c56c5646b-pkdsf"]
Jul 31 23:52:06.852742 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.852680    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c56c5646b-pkdsf" event={"ID":"361ef423-1058-4296-81de-f7df57bee743","Type":"ContainerStarted","Data":"4e61e027ab03f4f6dd59437f0238cc25c7b7fb228f42ee9cf1dae6502c20cc97"}
Jul 31 23:52:06.853185 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.852775    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c56c5646b-pkdsf" event={"ID":"361ef423-1058-4296-81de-f7df57bee743","Type":"ContainerStarted","Data":"5b793ddc0cba44c763ed0fb6cbf7596ca465fbe403a8d15b889ee0ecf57b5b37"}
Jul 31 23:52:06.857969 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.857942    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8"
Jul 31 23:52:06.857969 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.857953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbtt9v8" event={"ID":"4cbd7da5-9146-454c-946b-47caae76240f","Type":"ContainerDied","Data":"a91d87d411b9616e2e105434a88e6419359f94b6b7250ce40be6ae947d61380d"}
Jul 31 23:52:06.858176 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.857999    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a91d87d411b9616e2e105434a88e6419359f94b6b7250ce40be6ae947d61380d"
Jul 31 23:52:06.859953 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.859926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6" event={"ID":"943bb77e-5117-4b72-af58-76cbb352ac52","Type":"ContainerDied","Data":"4510ac3fc3e3455d873f08304ddf9218ea3c8c9a53d76864d66f45e7beb5947c"}
Jul 31 23:52:06.860077 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.859969    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4510ac3fc3e3455d873f08304ddf9218ea3c8c9a53d76864d66f45e7beb5947c"
Jul 31 23:52:06.860863 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.860327    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cfpvp6"
Jul 31 23:52:06.871351 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:06.871244    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c56c5646b-pkdsf" podStartSLOduration=1.871226769 podStartE2EDuration="1.871226769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:06.86893102 +0000 UTC m=+366.228822062" watchObservedRunningTime="2026-07-31 23:52:06.871226769 +0000 UTC m=+366.231117886"
Jul 31 23:52:07.016276 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.016250    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv"
Jul 31 23:52:07.019699 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.019682    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc"
Jul 31 23:52:07.123513 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.123424    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85chd\" (UniqueName: \"kubernetes.io/projected/015b3f73-2132-4d6f-b69e-0a77168b111d-kube-api-access-85chd\") pod \"015b3f73-2132-4d6f-b69e-0a77168b111d\" (UID: \"015b3f73-2132-4d6f-b69e-0a77168b111d\") "
Jul 31 23:52:07.123513 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.123478    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/015b3f73-2132-4d6f-b69e-0a77168b111d-bundle\") pod \"015b3f73-2132-4d6f-b69e-0a77168b111d\" (UID: \"015b3f73-2132-4d6f-b69e-0a77168b111d\") "
Jul 31 23:52:07.123763 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.123530    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48c3c95b-bbff-44f6-9d43-cb124ccaa9c8-util\") pod \"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8\" (UID: \"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8\") "
Jul 31 23:52:07.123763 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.123548    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48c3c95b-bbff-44f6-9d43-cb124ccaa9c8-bundle\") pod \"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8\" (UID: \"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8\") "
Jul 31 23:52:07.123763 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.123598    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6b8zs\" (UniqueName: \"kubernetes.io/projected/48c3c95b-bbff-44f6-9d43-cb124ccaa9c8-kube-api-access-6b8zs\") pod \"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8\" (UID: \"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8\") "
Jul 31 23:52:07.123763 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.123620    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/015b3f73-2132-4d6f-b69e-0a77168b111d-util\") pod \"015b3f73-2132-4d6f-b69e-0a77168b111d\" (UID: \"015b3f73-2132-4d6f-b69e-0a77168b111d\") "
Jul 31 23:52:07.124129 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.124099    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015b3f73-2132-4d6f-b69e-0a77168b111d-bundle" (OuterVolumeSpecName: "bundle") pod "015b3f73-2132-4d6f-b69e-0a77168b111d" (UID: "015b3f73-2132-4d6f-b69e-0a77168b111d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:52:07.124421 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.124353    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c3c95b-bbff-44f6-9d43-cb124ccaa9c8-bundle" (OuterVolumeSpecName: "bundle") pod "48c3c95b-bbff-44f6-9d43-cb124ccaa9c8" (UID: "48c3c95b-bbff-44f6-9d43-cb124ccaa9c8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:52:07.125921 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.125892    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c3c95b-bbff-44f6-9d43-cb124ccaa9c8-kube-api-access-6b8zs" (OuterVolumeSpecName: "kube-api-access-6b8zs") pod "48c3c95b-bbff-44f6-9d43-cb124ccaa9c8" (UID: "48c3c95b-bbff-44f6-9d43-cb124ccaa9c8"). InnerVolumeSpecName "kube-api-access-6b8zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:52:07.126030 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.125986    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015b3f73-2132-4d6f-b69e-0a77168b111d-kube-api-access-85chd" (OuterVolumeSpecName: "kube-api-access-85chd") pod "015b3f73-2132-4d6f-b69e-0a77168b111d" (UID: "015b3f73-2132-4d6f-b69e-0a77168b111d"). InnerVolumeSpecName "kube-api-access-85chd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:52:07.131355 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.131309    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c3c95b-bbff-44f6-9d43-cb124ccaa9c8-util" (OuterVolumeSpecName: "util") pod "48c3c95b-bbff-44f6-9d43-cb124ccaa9c8" (UID: "48c3c95b-bbff-44f6-9d43-cb124ccaa9c8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:52:07.132455 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.132420    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015b3f73-2132-4d6f-b69e-0a77168b111d-util" (OuterVolumeSpecName: "util") pod "015b3f73-2132-4d6f-b69e-0a77168b111d" (UID: "015b3f73-2132-4d6f-b69e-0a77168b111d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:52:07.224426 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.224387    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/015b3f73-2132-4d6f-b69e-0a77168b111d-util\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:07.224426 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.224417    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85chd\" (UniqueName: \"kubernetes.io/projected/015b3f73-2132-4d6f-b69e-0a77168b111d-kube-api-access-85chd\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:07.224426 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.224432    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/015b3f73-2132-4d6f-b69e-0a77168b111d-bundle\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:07.224642 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.224443    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48c3c95b-bbff-44f6-9d43-cb124ccaa9c8-util\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:07.224642 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.224454    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48c3c95b-bbff-44f6-9d43-cb124ccaa9c8-bundle\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:07.224642 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.224465    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b8zs\" (UniqueName: \"kubernetes.io/projected/48c3c95b-bbff-44f6-9d43-cb124ccaa9c8-kube-api-access-6b8zs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:07.867076 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.867047    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv"
Jul 31 23:52:07.867543 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.867043    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053g7dcv" event={"ID":"48c3c95b-bbff-44f6-9d43-cb124ccaa9c8","Type":"ContainerDied","Data":"c8f8461d2cec137d9d50fe3e71c1790d0a1606da71cda814f62e7f723ee408dc"}
Jul 31 23:52:07.867543 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.867166    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8f8461d2cec137d9d50fe3e71c1790d0a1606da71cda814f62e7f723ee408dc"
Jul 31 23:52:07.868995 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.868974    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc"
Jul 31 23:52:07.869133 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.868987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nzqkc" event={"ID":"015b3f73-2132-4d6f-b69e-0a77168b111d","Type":"ContainerDied","Data":"574f0adb1fa1a6349764fc9b688cef131f7759b88ebd56a34afda3f96e03cd87"}
Jul 31 23:52:07.869133 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:07.869043    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="574f0adb1fa1a6349764fc9b688cef131f7759b88ebd56a34afda3f96e03cd87"
Jul 31 23:52:12.625011 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.624972    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-9kq7d"]
Jul 31 23:52:12.625492 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625457    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="015b3f73-2132-4d6f-b69e-0a77168b111d" containerName="extract"
Jul 31 23:52:12.625492 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625474    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="015b3f73-2132-4d6f-b69e-0a77168b111d" containerName="extract"
Jul 31 23:52:12.625492 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625488    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="015b3f73-2132-4d6f-b69e-0a77168b111d" containerName="pull"
Jul 31 23:52:12.625657 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625496    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="015b3f73-2132-4d6f-b69e-0a77168b111d" containerName="pull"
Jul 31 23:52:12.625657 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625507    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="943bb77e-5117-4b72-af58-76cbb352ac52" containerName="pull"
Jul 31 23:52:12.625657 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625516    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="943bb77e-5117-4b72-af58-76cbb352ac52" containerName="pull"
Jul 31 23:52:12.625657 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625527    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="943bb77e-5117-4b72-af58-76cbb352ac52" containerName="extract"
Jul 31 23:52:12.625657 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625535    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="943bb77e-5117-4b72-af58-76cbb352ac52" containerName="extract"
Jul 31 23:52:12.625657 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625547    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cbd7da5-9146-454c-946b-47caae76240f" containerName="pull"
Jul 31 23:52:12.625657 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625554    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cbd7da5-9146-454c-946b-47caae76240f" containerName="pull"
Jul 31 23:52:12.625657 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625563    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cbd7da5-9146-454c-946b-47caae76240f" containerName="extract"
Jul 31 23:52:12.625657 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625571    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cbd7da5-9146-454c-946b-47caae76240f" containerName="extract"
Jul 31 23:52:12.625657 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625592    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c3c95b-bbff-44f6-9d43-cb124ccaa9c8" containerName="util"
Jul 31 23:52:12.625657 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625600    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c3c95b-bbff-44f6-9d43-cb124ccaa9c8" containerName="util"
Jul 31 23:52:12.625657 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625612    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="943bb77e-5117-4b72-af58-76cbb352ac52" containerName="util"
Jul 31 23:52:12.625657 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625621    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="943bb77e-5117-4b72-af58-76cbb352ac52" containerName="util"
Jul 31 23:52:12.625657 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625630    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c3c95b-bbff-44f6-9d43-cb124ccaa9c8" containerName="extract"
Jul 31 23:52:12.625657 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625639    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c3c95b-bbff-44f6-9d43-cb124ccaa9c8" containerName="extract"
Jul 31 23:52:12.625657 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625656    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cbd7da5-9146-454c-946b-47caae76240f" containerName="util"
Jul 31 23:52:12.625657 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625664    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cbd7da5-9146-454c-946b-47caae76240f" containerName="util"
Jul 31 23:52:12.626447 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625680    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="015b3f73-2132-4d6f-b69e-0a77168b111d" containerName="util"
Jul 31 23:52:12.626447 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625688    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="015b3f73-2132-4d6f-b69e-0a77168b111d" containerName="util"
Jul 31 23:52:12.626447 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625697    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c3c95b-bbff-44f6-9d43-cb124ccaa9c8" containerName="pull"
Jul 31 23:52:12.626447 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625704    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c3c95b-bbff-44f6-9d43-cb124ccaa9c8" containerName="pull"
Jul 31 23:52:12.626447 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625807    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cbd7da5-9146-454c-946b-47caae76240f" containerName="extract"
Jul 31 23:52:12.626447 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625820    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="943bb77e-5117-4b72-af58-76cbb352ac52" containerName="extract"
Jul 31 23:52:12.626447 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625831    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="48c3c95b-bbff-44f6-9d43-cb124ccaa9c8" containerName="extract"
Jul 31 23:52:12.626447 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.625839    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="015b3f73-2132-4d6f-b69e-0a77168b111d" containerName="extract"
Jul 31 23:52:12.631011 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.630992    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9kq7d"
Jul 31 23:52:12.633211 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.633192    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 31 23:52:12.633341 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.633284    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 31 23:52:12.633586 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.633563    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 31 23:52:12.633586 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.633584    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-xs6bj\""
Jul 31 23:52:12.642886 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.642864    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-9kq7d"]
Jul 31 23:52:12.777416 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.777386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c6nk\" (UniqueName: \"kubernetes.io/projected/1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26-kube-api-access-2c6nk\") pod \"dns-operator-controller-manager-ccbdf686-9kq7d\" (UID: \"1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9kq7d"
Jul 31 23:52:12.878097 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.878007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c6nk\" (UniqueName: \"kubernetes.io/projected/1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26-kube-api-access-2c6nk\") pod \"dns-operator-controller-manager-ccbdf686-9kq7d\" (UID: \"1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9kq7d"
Jul 31 23:52:12.887075 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.887045    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c6nk\" (UniqueName: \"kubernetes.io/projected/1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26-kube-api-access-2c6nk\") pod \"dns-operator-controller-manager-ccbdf686-9kq7d\" (UID: \"1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9kq7d"
Jul 31 23:52:12.941831 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:12.941798    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9kq7d"
Jul 31 23:52:13.102770 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:13.102741    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-9kq7d"]
Jul 31 23:52:13.106705 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:52:13.106679    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1ba4de_4d45_4d0c_b69b_1fbfa3adee26.slice/crio-a5fd499917fb581d40fba496a45b2303dcb5178415585d5b7128773ad4d1c42f WatchSource:0}: Error finding container a5fd499917fb581d40fba496a45b2303dcb5178415585d5b7128773ad4d1c42f: Status 404 returned error can't find the container with id a5fd499917fb581d40fba496a45b2303dcb5178415585d5b7128773ad4d1c42f
Jul 31 23:52:13.899130 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:13.899090    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9kq7d" event={"ID":"1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26","Type":"ContainerStarted","Data":"a5fd499917fb581d40fba496a45b2303dcb5178415585d5b7128773ad4d1c42f"}
Jul 31 23:52:15.197077 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:15.197046    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-9ds2s"]
Jul 31 23:52:15.206854 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:15.206821    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-9ds2s"
Jul 31 23:52:15.209198 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:15.209171    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-qllb7\""
Jul 31 23:52:15.211768 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:15.211686    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-9ds2s"]
Jul 31 23:52:15.308550 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:15.308515    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trtqw\" (UniqueName: \"kubernetes.io/projected/01d872a4-52b1-48d5-ab18-45fbd55718cd-kube-api-access-trtqw\") pod \"authorino-operator-56fb6f6856-9ds2s\" (UID: \"01d872a4-52b1-48d5-ab18-45fbd55718cd\") " pod="kuadrant-system/authorino-operator-56fb6f6856-9ds2s"
Jul 31 23:52:15.409236 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:15.409199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trtqw\" (UniqueName: \"kubernetes.io/projected/01d872a4-52b1-48d5-ab18-45fbd55718cd-kube-api-access-trtqw\") pod \"authorino-operator-56fb6f6856-9ds2s\" (UID: \"01d872a4-52b1-48d5-ab18-45fbd55718cd\") " pod="kuadrant-system/authorino-operator-56fb6f6856-9ds2s"
Jul 31 23:52:15.421646 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:15.421613    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trtqw\" (UniqueName: \"kubernetes.io/projected/01d872a4-52b1-48d5-ab18-45fbd55718cd-kube-api-access-trtqw\") pod \"authorino-operator-56fb6f6856-9ds2s\" (UID: \"01d872a4-52b1-48d5-ab18-45fbd55718cd\") " pod="kuadrant-system/authorino-operator-56fb6f6856-9ds2s"
Jul 31 23:52:15.521328 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:15.521243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-9ds2s"
Jul 31 23:52:15.921757 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:15.921700    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-9ds2s"]
Jul 31 23:52:15.924605 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:52:15.924582    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d872a4_52b1_48d5_ab18_45fbd55718cd.slice/crio-117c2e4fbba43d6f5260e026db61ce26f2b81a04e46650b7b450c5d2726aee2f WatchSource:0}: Error finding container 117c2e4fbba43d6f5260e026db61ce26f2b81a04e46650b7b450c5d2726aee2f: Status 404 returned error can't find the container with id 117c2e4fbba43d6f5260e026db61ce26f2b81a04e46650b7b450c5d2726aee2f
Jul 31 23:52:16.205236 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:16.205203    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:16.205624 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:16.205261    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:16.210515 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:16.210492    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:16.918784 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:16.918745    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-9ds2s" event={"ID":"01d872a4-52b1-48d5-ab18-45fbd55718cd","Type":"ContainerStarted","Data":"117c2e4fbba43d6f5260e026db61ce26f2b81a04e46650b7b450c5d2726aee2f"}
Jul 31 23:52:16.920752 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:16.920552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9kq7d" event={"ID":"1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26","Type":"ContainerStarted","Data":"1dbc464fe87bfe885b3844d3c14e34152a1e739856eb4a67992e64d61874bd13"}
Jul 31 23:52:16.920752 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:16.920631    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9kq7d"
Jul 31 23:52:16.925109 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:16.925080    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c56c5646b-pkdsf"
Jul 31 23:52:16.945022 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:16.944978    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9kq7d" podStartSLOduration=2.214193803 podStartE2EDuration="4.94496234s" totalImagesPullingTime="2.730768537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:16.943535704 +0000 UTC m=+376.303426750" watchObservedRunningTime="2026-07-31 23:52:16.94496234 +0000 UTC m=+376.304853380"
Jul 31 23:52:16.991417 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:16.991384    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fd89469fd-96jjr"]
Jul 31 23:52:17.509871 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:17.509836    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-2w49s"]
Jul 31 23:52:17.513448 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:17.513426    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-2w49s"
Jul 31 23:52:17.515271 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:17.515251    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-q6ll7\""
Jul 31 23:52:17.527588 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:17.527558    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-2w49s"]
Jul 31 23:52:17.632432 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:17.632401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrwrh\" (UniqueName: \"kubernetes.io/projected/96ef8b85-f389-4d7f-916b-cc17bb35cf2f-kube-api-access-nrwrh\") pod \"limitador-operator-controller-manager-774d94476f-2w49s\" (UID: \"96ef8b85-f389-4d7f-916b-cc17bb35cf2f\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-2w49s"
Jul 31 23:52:17.733730 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:17.733677    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrwrh\" (UniqueName: \"kubernetes.io/projected/96ef8b85-f389-4d7f-916b-cc17bb35cf2f-kube-api-access-nrwrh\") pod \"limitador-operator-controller-manager-774d94476f-2w49s\" (UID: \"96ef8b85-f389-4d7f-916b-cc17bb35cf2f\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-2w49s"
Jul 31 23:52:17.741937 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:17.741904    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrwrh\" (UniqueName: \"kubernetes.io/projected/96ef8b85-f389-4d7f-916b-cc17bb35cf2f-kube-api-access-nrwrh\") pod \"limitador-operator-controller-manager-774d94476f-2w49s\" (UID: \"96ef8b85-f389-4d7f-916b-cc17bb35cf2f\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-2w49s"
Jul 31 23:52:17.823644 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:17.823564    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-2w49s"
Jul 31 23:52:17.928933 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:17.928815    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-9ds2s" event={"ID":"01d872a4-52b1-48d5-ab18-45fbd55718cd","Type":"ContainerStarted","Data":"dd98ecdb06bee724e199d8cb1754a64cf3931e147399cf040c0cfbe7e8267deb"}
Jul 31 23:52:17.955821 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:17.955767    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-56fb6f6856-9ds2s" podStartSLOduration=1.340435823 podStartE2EDuration="2.955750192s" totalImagesPullingTime="1.615314369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:17.954357047 +0000 UTC m=+377.314248109" watchObservedRunningTime="2026-07-31 23:52:17.955750192 +0000 UTC m=+377.315641251"
Jul 31 23:52:17.967342 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:17.967321    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-2w49s"]
Jul 31 23:52:17.969409 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:52:17.969383    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ef8b85_f389_4d7f_916b_cc17bb35cf2f.slice/crio-af8ee8523098d3f36b81827035999f514737a8851fe039f6f6e2f8dff046c8f4 WatchSource:0}: Error finding container af8ee8523098d3f36b81827035999f514737a8851fe039f6f6e2f8dff046c8f4: Status 404 returned error can't find the container with id af8ee8523098d3f36b81827035999f514737a8851fe039f6f6e2f8dff046c8f4
Jul 31 23:52:18.935112 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:18.935078    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-2w49s" event={"ID":"96ef8b85-f389-4d7f-916b-cc17bb35cf2f","Type":"ContainerStarted","Data":"af8ee8523098d3f36b81827035999f514737a8851fe039f6f6e2f8dff046c8f4"}
Jul 31 23:52:18.935504 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:18.935257    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-56fb6f6856-9ds2s"
Jul 31 23:52:20.948324 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:20.948281    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-2w49s" event={"ID":"96ef8b85-f389-4d7f-916b-cc17bb35cf2f","Type":"ContainerStarted","Data":"bd525e380bd65dc15c1a0cf55cbba92ebd8a0c8d1a369ddd464fcedd8da31818"}
Jul 31 23:52:20.948706 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:20.948429    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-2w49s"
Jul 31 23:52:20.968647 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:20.968595    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-2w49s" podStartSLOduration=1.563309309 podStartE2EDuration="3.968580624s" totalImagesPullingTime="2.405271315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:20.966302164 +0000 UTC m=+380.326193222" watchObservedRunningTime="2026-07-31 23:52:20.968580624 +0000 UTC m=+380.328471698"
Jul 31 23:52:21.442781 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:21.442740    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-7qwtd"]
Jul 31 23:52:21.446626 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:21.446603    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-7qwtd"
Jul 31 23:52:21.448809 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:21.448790    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8w9k2\""
Jul 31 23:52:21.463759 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:21.463729    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-7qwtd"]
Jul 31 23:52:21.568284 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:21.568238    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5czvm\" (UniqueName: \"kubernetes.io/projected/33532000-c2e7-4881-ac4b-d27d581dbdad-kube-api-access-5czvm\") pod \"kuadrant-operator-controller-manager-66866f5fcc-7qwtd\" (UID: \"33532000-c2e7-4881-ac4b-d27d581dbdad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-7qwtd"
Jul 31 23:52:21.568463 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:21.568309    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33532000-c2e7-4881-ac4b-d27d581dbdad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-7qwtd\" (UID: \"33532000-c2e7-4881-ac4b-d27d581dbdad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-7qwtd"
Jul 31 23:52:21.669576 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:21.669543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5czvm\" (UniqueName: \"kubernetes.io/projected/33532000-c2e7-4881-ac4b-d27d581dbdad-kube-api-access-5czvm\") pod \"kuadrant-operator-controller-manager-66866f5fcc-7qwtd\" (UID: \"33532000-c2e7-4881-ac4b-d27d581dbdad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-7qwtd"
Jul 31 23:52:21.669768 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:21.669609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33532000-c2e7-4881-ac4b-d27d581dbdad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-7qwtd\" (UID: \"33532000-c2e7-4881-ac4b-d27d581dbdad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-7qwtd"
Jul 31 23:52:21.670054 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:21.670034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33532000-c2e7-4881-ac4b-d27d581dbdad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-7qwtd\" (UID: \"33532000-c2e7-4881-ac4b-d27d581dbdad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-7qwtd"
Jul 31 23:52:21.677730 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:21.677690    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5czvm\" (UniqueName: \"kubernetes.io/projected/33532000-c2e7-4881-ac4b-d27d581dbdad-kube-api-access-5czvm\") pod \"kuadrant-operator-controller-manager-66866f5fcc-7qwtd\" (UID: \"33532000-c2e7-4881-ac4b-d27d581dbdad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-7qwtd"
Jul 31 23:52:21.756543 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:21.756456    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-7qwtd"
Jul 31 23:52:21.895902 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:21.895879    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-7qwtd"]
Jul 31 23:52:21.898415 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:52:21.898385    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33532000_c2e7_4881_ac4b_d27d581dbdad.slice/crio-49dbc24c14a0a2e2c4560773a15fb144f481ef87c7214d296342427b6485f2db WatchSource:0}: Error finding container 49dbc24c14a0a2e2c4560773a15fb144f481ef87c7214d296342427b6485f2db: Status 404 returned error can't find the container with id 49dbc24c14a0a2e2c4560773a15fb144f481ef87c7214d296342427b6485f2db
Jul 31 23:52:21.954671 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:21.954638    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-7qwtd" event={"ID":"33532000-c2e7-4881-ac4b-d27d581dbdad","Type":"ContainerStarted","Data":"49dbc24c14a0a2e2c4560773a15fb144f481ef87c7214d296342427b6485f2db"}
Jul 31 23:52:27.931623 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:27.931593    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9kq7d"
Jul 31 23:52:27.998602 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:27.998565    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-7qwtd" event={"ID":"33532000-c2e7-4881-ac4b-d27d581dbdad","Type":"ContainerStarted","Data":"399272d25933a787c3244395a25b0cb1618e9c36f497d2322c5e662791117ddd"}
Jul 31 23:52:27.998810 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:27.998700    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-7qwtd"
Jul 31 23:52:28.018019 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:28.017965    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-7qwtd" podStartSLOduration=1.580876664 podStartE2EDuration="7.017948907s" totalImagesPullingTime="5.437072243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:28.015371211 +0000 UTC m=+387.375262252" watchObservedRunningTime="2026-07-31 23:52:28.017948907 +0000 UTC m=+387.377839947"
Jul 31 23:52:29.943686 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:29.943660    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-56fb6f6856-9ds2s"
Jul 31 23:52:31.957006 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:31.956973    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-2w49s"
Jul 31 23:52:39.007021 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:39.006992    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-7qwtd"
Jul 31 23:52:42.013162 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.013102    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-fd89469fd-96jjr" podUID="61f880b5-8369-40fb-969c-c54fa77f2c59" containerName="console" containerID="cri-o://90bce9c9be40a163c6aa4b165970f6d091b115e2e534fbcc67d5d73d828aba04" gracePeriod=15
Jul 31 23:52:42.251382 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.251356    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:52:42.341706 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.341620    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-trusted-ca-bundle\") pod \"61f880b5-8369-40fb-969c-c54fa77f2c59\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") "
Jul 31 23:52:42.341706 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.341687    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61f880b5-8369-40fb-969c-c54fa77f2c59-console-serving-cert\") pod \"61f880b5-8369-40fb-969c-c54fa77f2c59\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") "
Jul 31 23:52:42.341706 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.341708    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-console-config\") pod \"61f880b5-8369-40fb-969c-c54fa77f2c59\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") "
Jul 31 23:52:42.342022 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.341768    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61f880b5-8369-40fb-969c-c54fa77f2c59-console-oauth-config\") pod \"61f880b5-8369-40fb-969c-c54fa77f2c59\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") "
Jul 31 23:52:42.342022 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.341786    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-oauth-serving-cert\") pod \"61f880b5-8369-40fb-969c-c54fa77f2c59\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") "
Jul 31 23:52:42.342022 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.341805    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-service-ca\") pod \"61f880b5-8369-40fb-969c-c54fa77f2c59\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") "
Jul 31 23:52:42.342022 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.341846    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45cqr\" (UniqueName: \"kubernetes.io/projected/61f880b5-8369-40fb-969c-c54fa77f2c59-kube-api-access-45cqr\") pod \"61f880b5-8369-40fb-969c-c54fa77f2c59\" (UID: \"61f880b5-8369-40fb-969c-c54fa77f2c59\") "
Jul 31 23:52:42.342235 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.342180    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "61f880b5-8369-40fb-969c-c54fa77f2c59" (UID: "61f880b5-8369-40fb-969c-c54fa77f2c59"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:52:42.342235 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.342211    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "61f880b5-8369-40fb-969c-c54fa77f2c59" (UID: "61f880b5-8369-40fb-969c-c54fa77f2c59"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:52:42.342407 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.342249    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-console-config" (OuterVolumeSpecName: "console-config") pod "61f880b5-8369-40fb-969c-c54fa77f2c59" (UID: "61f880b5-8369-40fb-969c-c54fa77f2c59"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:52:42.342407 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.342289    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-service-ca" (OuterVolumeSpecName: "service-ca") pod "61f880b5-8369-40fb-969c-c54fa77f2c59" (UID: "61f880b5-8369-40fb-969c-c54fa77f2c59"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:52:42.344081 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.344049    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f880b5-8369-40fb-969c-c54fa77f2c59-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "61f880b5-8369-40fb-969c-c54fa77f2c59" (UID: "61f880b5-8369-40fb-969c-c54fa77f2c59"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:52:42.344183 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.344102    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f880b5-8369-40fb-969c-c54fa77f2c59-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "61f880b5-8369-40fb-969c-c54fa77f2c59" (UID: "61f880b5-8369-40fb-969c-c54fa77f2c59"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:52:42.344183 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.344115    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f880b5-8369-40fb-969c-c54fa77f2c59-kube-api-access-45cqr" (OuterVolumeSpecName: "kube-api-access-45cqr") pod "61f880b5-8369-40fb-969c-c54fa77f2c59" (UID: "61f880b5-8369-40fb-969c-c54fa77f2c59"). InnerVolumeSpecName "kube-api-access-45cqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:52:42.443350 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.443323    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-oauth-serving-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:42.443350 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.443347    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-service-ca\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:42.443350 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.443356    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45cqr\" (UniqueName: \"kubernetes.io/projected/61f880b5-8369-40fb-969c-c54fa77f2c59-kube-api-access-45cqr\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:42.443592 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.443365    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-trusted-ca-bundle\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:42.443592 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.443375    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61f880b5-8369-40fb-969c-c54fa77f2c59-console-serving-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:42.443592 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.443383    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61f880b5-8369-40fb-969c-c54fa77f2c59-console-config\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:42.443592 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:42.443394    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61f880b5-8369-40fb-969c-c54fa77f2c59-console-oauth-config\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:52:43.079308 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:43.079271    2578 generic.go:356] "Generic (PLEG): container finished" podID="61f880b5-8369-40fb-969c-c54fa77f2c59" containerID="90bce9c9be40a163c6aa4b165970f6d091b115e2e534fbcc67d5d73d828aba04" exitCode=0
Jul 31 23:52:43.079818 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:43.079322    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd89469fd-96jjr"
Jul 31 23:52:43.079818 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:43.079360    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd89469fd-96jjr" event={"ID":"61f880b5-8369-40fb-969c-c54fa77f2c59","Type":"ContainerDied","Data":"90bce9c9be40a163c6aa4b165970f6d091b115e2e534fbcc67d5d73d828aba04"}
Jul 31 23:52:43.079818 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:43.079410    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd89469fd-96jjr" event={"ID":"61f880b5-8369-40fb-969c-c54fa77f2c59","Type":"ContainerDied","Data":"eb154ea56ad191e1d763fc9fd8da6605dfc321d30287709c15d7072acd19665e"}
Jul 31 23:52:43.079818 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:43.079429    2578 scope.go:117] "RemoveContainer" containerID="90bce9c9be40a163c6aa4b165970f6d091b115e2e534fbcc67d5d73d828aba04"
Jul 31 23:52:43.089146 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:43.089129    2578 scope.go:117] "RemoveContainer" containerID="90bce9c9be40a163c6aa4b165970f6d091b115e2e534fbcc67d5d73d828aba04"
Jul 31 23:52:43.089392 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:52:43.089376    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90bce9c9be40a163c6aa4b165970f6d091b115e2e534fbcc67d5d73d828aba04\": container with ID starting with 90bce9c9be40a163c6aa4b165970f6d091b115e2e534fbcc67d5d73d828aba04 not found: ID does not exist" containerID="90bce9c9be40a163c6aa4b165970f6d091b115e2e534fbcc67d5d73d828aba04"
Jul 31 23:52:43.089433 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:43.089399    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90bce9c9be40a163c6aa4b165970f6d091b115e2e534fbcc67d5d73d828aba04"} err="failed to get container status \"90bce9c9be40a163c6aa4b165970f6d091b115e2e534fbcc67d5d73d828aba04\": rpc error: code = NotFound desc = could not find container \"90bce9c9be40a163c6aa4b165970f6d091b115e2e534fbcc67d5d73d828aba04\": container with ID starting with 90bce9c9be40a163c6aa4b165970f6d091b115e2e534fbcc67d5d73d828aba04 not found: ID does not exist"
Jul 31 23:52:43.100900 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:43.100879    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fd89469fd-96jjr"]
Jul 31 23:52:43.111048 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:43.111027    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fd89469fd-96jjr"]
Jul 31 23:52:43.185708 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:52:43.185676    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61f880b5-8369-40fb-969c-c54fa77f2c59" path="/var/lib/kubelet/pods/61f880b5-8369-40fb-969c-c54fa77f2c59/volumes"
Jul 31 23:53:14.081861 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.081830    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-mkxsc"]
Jul 31 23:53:14.082302 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.082180    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61f880b5-8369-40fb-969c-c54fa77f2c59" containerName="console"
Jul 31 23:53:14.082302 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.082192    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="61f880b5-8369-40fb-969c-c54fa77f2c59" containerName="console"
Jul 31 23:53:14.082302 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.082254    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="61f880b5-8369-40fb-969c-c54fa77f2c59" containerName="console"
Jul 31 23:53:14.085857 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.085835    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-mkxsc"
Jul 31 23:53:14.087896 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.087876    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xmj24\""
Jul 31 23:53:14.099345 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.099319    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-mkxsc"]
Jul 31 23:53:14.212122 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.212085    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtxj7\" (UniqueName: \"kubernetes.io/projected/5a73afb8-5a54-4520-9f83-5cf0d24e434b-kube-api-access-mtxj7\") pod \"authorino-b54df94b4-mkxsc\" (UID: \"5a73afb8-5a54-4520-9f83-5cf0d24e434b\") " pod="kuadrant-system/authorino-b54df94b4-mkxsc"
Jul 31 23:53:14.312745 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.312681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtxj7\" (UniqueName: \"kubernetes.io/projected/5a73afb8-5a54-4520-9f83-5cf0d24e434b-kube-api-access-mtxj7\") pod \"authorino-b54df94b4-mkxsc\" (UID: \"5a73afb8-5a54-4520-9f83-5cf0d24e434b\") " pod="kuadrant-system/authorino-b54df94b4-mkxsc"
Jul 31 23:53:14.314906 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.314883    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-65flf"]
Jul 31 23:53:14.317810 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.317794    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-65flf"
Jul 31 23:53:14.321190 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.321168    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtxj7\" (UniqueName: \"kubernetes.io/projected/5a73afb8-5a54-4520-9f83-5cf0d24e434b-kube-api-access-mtxj7\") pod \"authorino-b54df94b4-mkxsc\" (UID: \"5a73afb8-5a54-4520-9f83-5cf0d24e434b\") " pod="kuadrant-system/authorino-b54df94b4-mkxsc"
Jul 31 23:53:14.324508 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.324487    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-65flf"]
Jul 31 23:53:14.395076 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.395045    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-mkxsc"
Jul 31 23:53:14.413968 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.413940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw4zv\" (UniqueName: \"kubernetes.io/projected/76aec74b-50a2-48fb-9412-1b64cec4ec3b-kube-api-access-dw4zv\") pod \"authorino-c548b5d7d-65flf\" (UID: \"76aec74b-50a2-48fb-9412-1b64cec4ec3b\") " pod="kuadrant-system/authorino-c548b5d7d-65flf"
Jul 31 23:53:14.514738 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.514688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw4zv\" (UniqueName: \"kubernetes.io/projected/76aec74b-50a2-48fb-9412-1b64cec4ec3b-kube-api-access-dw4zv\") pod \"authorino-c548b5d7d-65flf\" (UID: \"76aec74b-50a2-48fb-9412-1b64cec4ec3b\") " pod="kuadrant-system/authorino-c548b5d7d-65flf"
Jul 31 23:53:14.522253 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.522228    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw4zv\" (UniqueName: \"kubernetes.io/projected/76aec74b-50a2-48fb-9412-1b64cec4ec3b-kube-api-access-dw4zv\") pod \"authorino-c548b5d7d-65flf\" (UID: \"76aec74b-50a2-48fb-9412-1b64cec4ec3b\") " pod="kuadrant-system/authorino-c548b5d7d-65flf"
Jul 31 23:53:14.534552 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.534524    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-mkxsc"]
Jul 31 23:53:14.537707 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:53:14.537671    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a73afb8_5a54_4520_9f83_5cf0d24e434b.slice/crio-3aa011e07b1178866b4a2f090e44062b5dc962ffd263cce76a07385024abe581 WatchSource:0}: Error finding container 3aa011e07b1178866b4a2f090e44062b5dc962ffd263cce76a07385024abe581: Status 404 returned error can't find the container with id 3aa011e07b1178866b4a2f090e44062b5dc962ffd263cce76a07385024abe581
Jul 31 23:53:14.637344 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.637250    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-65flf"
Jul 31 23:53:14.775162 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:14.775135    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-65flf"]
Jul 31 23:53:14.776918 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:53:14.776891    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76aec74b_50a2_48fb_9412_1b64cec4ec3b.slice/crio-0b0280310870117bc8f44f9d821d7b411590e88dba0741ee492d11e9fa916617 WatchSource:0}: Error finding container 0b0280310870117bc8f44f9d821d7b411590e88dba0741ee492d11e9fa916617: Status 404 returned error can't find the container with id 0b0280310870117bc8f44f9d821d7b411590e88dba0741ee492d11e9fa916617
Jul 31 23:53:15.244820 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:15.244788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-mkxsc" event={"ID":"5a73afb8-5a54-4520-9f83-5cf0d24e434b","Type":"ContainerStarted","Data":"3aa011e07b1178866b4a2f090e44062b5dc962ffd263cce76a07385024abe581"}
Jul 31 23:53:15.246239 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:15.246208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-65flf" event={"ID":"76aec74b-50a2-48fb-9412-1b64cec4ec3b","Type":"ContainerStarted","Data":"0b0280310870117bc8f44f9d821d7b411590e88dba0741ee492d11e9fa916617"}
Jul 31 23:53:17.263195 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:17.263153    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-mkxsc" event={"ID":"5a73afb8-5a54-4520-9f83-5cf0d24e434b","Type":"ContainerStarted","Data":"42208b5b6931cb7e2842d0643383f9e8d93996e25901fc8692d675f675f19617"}
Jul 31 23:53:17.264638 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:17.264601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-65flf" event={"ID":"76aec74b-50a2-48fb-9412-1b64cec4ec3b","Type":"ContainerStarted","Data":"cb7cf8f3c5a85413aeb17716f94be83420504b8ffd12da23a718f7e8c981e47f"}
Jul 31 23:53:17.279257 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:17.279205    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-mkxsc" podStartSLOduration=1.102654154 podStartE2EDuration="3.279189837s" totalImagesPullingTime="2.176535683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:53:17.276615744 +0000 UTC m=+436.636506785" watchObservedRunningTime="2026-07-31 23:53:17.279189837 +0000 UTC m=+436.639080878"
Jul 31 23:53:17.290051 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:17.289634    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-65flf" podStartSLOduration=1.3412482190000001 podStartE2EDuration="3.289620246s" totalImagesPullingTime="1.948372027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:53:17.289422088 +0000 UTC m=+436.649313130" watchObservedRunningTime="2026-07-31 23:53:17.289620246 +0000 UTC m=+436.649511288"
Jul 31 23:53:17.317376 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:17.317347    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-mkxsc"]
Jul 31 23:53:19.275123 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:19.275069    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-mkxsc" podUID="5a73afb8-5a54-4520-9f83-5cf0d24e434b" containerName="authorino" containerID="cri-o://42208b5b6931cb7e2842d0643383f9e8d93996e25901fc8692d675f675f19617" gracePeriod=30
Jul 31 23:53:19.522504 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:19.522480    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-mkxsc"
Jul 31 23:53:19.662499 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:19.662460    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtxj7\" (UniqueName: \"kubernetes.io/projected/5a73afb8-5a54-4520-9f83-5cf0d24e434b-kube-api-access-mtxj7\") pod \"5a73afb8-5a54-4520-9f83-5cf0d24e434b\" (UID: \"5a73afb8-5a54-4520-9f83-5cf0d24e434b\") "
Jul 31 23:53:19.664783 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:19.664758    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a73afb8-5a54-4520-9f83-5cf0d24e434b-kube-api-access-mtxj7" (OuterVolumeSpecName: "kube-api-access-mtxj7") pod "5a73afb8-5a54-4520-9f83-5cf0d24e434b" (UID: "5a73afb8-5a54-4520-9f83-5cf0d24e434b"). InnerVolumeSpecName "kube-api-access-mtxj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:53:19.763324 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:19.763289    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtxj7\" (UniqueName: \"kubernetes.io/projected/5a73afb8-5a54-4520-9f83-5cf0d24e434b-kube-api-access-mtxj7\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:53:20.281794 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:20.281761    2578 generic.go:356] "Generic (PLEG): container finished" podID="5a73afb8-5a54-4520-9f83-5cf0d24e434b" containerID="42208b5b6931cb7e2842d0643383f9e8d93996e25901fc8692d675f675f19617" exitCode=0
Jul 31 23:53:20.282259 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:20.281804    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-mkxsc"
Jul 31 23:53:20.282259 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:20.281843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-mkxsc" event={"ID":"5a73afb8-5a54-4520-9f83-5cf0d24e434b","Type":"ContainerDied","Data":"42208b5b6931cb7e2842d0643383f9e8d93996e25901fc8692d675f675f19617"}
Jul 31 23:53:20.282259 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:20.281894    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-mkxsc" event={"ID":"5a73afb8-5a54-4520-9f83-5cf0d24e434b","Type":"ContainerDied","Data":"3aa011e07b1178866b4a2f090e44062b5dc962ffd263cce76a07385024abe581"}
Jul 31 23:53:20.282259 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:20.281919    2578 scope.go:117] "RemoveContainer" containerID="42208b5b6931cb7e2842d0643383f9e8d93996e25901fc8692d675f675f19617"
Jul 31 23:53:20.290915 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:20.290894    2578 scope.go:117] "RemoveContainer" containerID="42208b5b6931cb7e2842d0643383f9e8d93996e25901fc8692d675f675f19617"
Jul 31 23:53:20.291185 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:53:20.291170    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42208b5b6931cb7e2842d0643383f9e8d93996e25901fc8692d675f675f19617\": container with ID starting with 42208b5b6931cb7e2842d0643383f9e8d93996e25901fc8692d675f675f19617 not found: ID does not exist" containerID="42208b5b6931cb7e2842d0643383f9e8d93996e25901fc8692d675f675f19617"
Jul 31 23:53:20.291227 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:20.291194    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42208b5b6931cb7e2842d0643383f9e8d93996e25901fc8692d675f675f19617"} err="failed to get container status \"42208b5b6931cb7e2842d0643383f9e8d93996e25901fc8692d675f675f19617\": rpc error: code = NotFound desc = could not find container \"42208b5b6931cb7e2842d0643383f9e8d93996e25901fc8692d675f675f19617\": container with ID starting with 42208b5b6931cb7e2842d0643383f9e8d93996e25901fc8692d675f675f19617 not found: ID does not exist"
Jul 31 23:53:20.302787 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:20.302760    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-mkxsc"]
Jul 31 23:53:20.306218 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:20.306195    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-mkxsc"]
Jul 31 23:53:21.184915 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:21.184884    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a73afb8-5a54-4520-9f83-5cf0d24e434b" path="/var/lib/kubelet/pods/5a73afb8-5a54-4520-9f83-5cf0d24e434b/volumes"
Jul 31 23:53:38.852319 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:38.852281    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-65flf"]
Jul 31 23:53:38.852886 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:38.852476    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-65flf" podUID="76aec74b-50a2-48fb-9412-1b64cec4ec3b" containerName="authorino" containerID="cri-o://cb7cf8f3c5a85413aeb17716f94be83420504b8ffd12da23a718f7e8c981e47f" gracePeriod=30
Jul 31 23:53:39.113008 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:39.112955    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-65flf"
Jul 31 23:53:39.232623 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:39.232589    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw4zv\" (UniqueName: \"kubernetes.io/projected/76aec74b-50a2-48fb-9412-1b64cec4ec3b-kube-api-access-dw4zv\") pod \"76aec74b-50a2-48fb-9412-1b64cec4ec3b\" (UID: \"76aec74b-50a2-48fb-9412-1b64cec4ec3b\") "
Jul 31 23:53:39.234765 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:39.234739    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76aec74b-50a2-48fb-9412-1b64cec4ec3b-kube-api-access-dw4zv" (OuterVolumeSpecName: "kube-api-access-dw4zv") pod "76aec74b-50a2-48fb-9412-1b64cec4ec3b" (UID: "76aec74b-50a2-48fb-9412-1b64cec4ec3b"). InnerVolumeSpecName "kube-api-access-dw4zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:53:39.334092 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:39.334063    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw4zv\" (UniqueName: \"kubernetes.io/projected/76aec74b-50a2-48fb-9412-1b64cec4ec3b-kube-api-access-dw4zv\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:53:39.390123 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:39.390028    2578 generic.go:356] "Generic (PLEG): container finished" podID="76aec74b-50a2-48fb-9412-1b64cec4ec3b" containerID="cb7cf8f3c5a85413aeb17716f94be83420504b8ffd12da23a718f7e8c981e47f" exitCode=0
Jul 31 23:53:39.390123 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:39.390072    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-65flf"
Jul 31 23:53:39.390123 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:39.390098    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-65flf" event={"ID":"76aec74b-50a2-48fb-9412-1b64cec4ec3b","Type":"ContainerDied","Data":"cb7cf8f3c5a85413aeb17716f94be83420504b8ffd12da23a718f7e8c981e47f"}
Jul 31 23:53:39.390399 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:39.390135    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-65flf" event={"ID":"76aec74b-50a2-48fb-9412-1b64cec4ec3b","Type":"ContainerDied","Data":"0b0280310870117bc8f44f9d821d7b411590e88dba0741ee492d11e9fa916617"}
Jul 31 23:53:39.390399 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:39.390152    2578 scope.go:117] "RemoveContainer" containerID="cb7cf8f3c5a85413aeb17716f94be83420504b8ffd12da23a718f7e8c981e47f"
Jul 31 23:53:39.399832 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:39.399810    2578 scope.go:117] "RemoveContainer" containerID="cb7cf8f3c5a85413aeb17716f94be83420504b8ffd12da23a718f7e8c981e47f"
Jul 31 23:53:39.400082 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:53:39.400065    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7cf8f3c5a85413aeb17716f94be83420504b8ffd12da23a718f7e8c981e47f\": container with ID starting with cb7cf8f3c5a85413aeb17716f94be83420504b8ffd12da23a718f7e8c981e47f not found: ID does not exist" containerID="cb7cf8f3c5a85413aeb17716f94be83420504b8ffd12da23a718f7e8c981e47f"
Jul 31 23:53:39.400128 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:39.400090    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7cf8f3c5a85413aeb17716f94be83420504b8ffd12da23a718f7e8c981e47f"} err="failed to get container status \"cb7cf8f3c5a85413aeb17716f94be83420504b8ffd12da23a718f7e8c981e47f\": rpc error: code = NotFound desc = could not find container \"cb7cf8f3c5a85413aeb17716f94be83420504b8ffd12da23a718f7e8c981e47f\": container with ID starting with cb7cf8f3c5a85413aeb17716f94be83420504b8ffd12da23a718f7e8c981e47f not found: ID does not exist"
Jul 31 23:53:39.414549 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:39.414513    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-65flf"]
Jul 31 23:53:39.416329 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:39.416307    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-65flf"]
Jul 31 23:53:41.185492 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:41.185458    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76aec74b-50a2-48fb-9412-1b64cec4ec3b" path="/var/lib/kubelet/pods/76aec74b-50a2-48fb-9412-1b64cec4ec3b/volumes"
Jul 31 23:53:55.354298 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.354260    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"]
Jul 31 23:53:55.354700 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.354613    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a73afb8-5a54-4520-9f83-5cf0d24e434b" containerName="authorino"
Jul 31 23:53:55.354700 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.354624    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a73afb8-5a54-4520-9f83-5cf0d24e434b" containerName="authorino"
Jul 31 23:53:55.354700 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.354642    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76aec74b-50a2-48fb-9412-1b64cec4ec3b" containerName="authorino"
Jul 31 23:53:55.354700 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.354647    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="76aec74b-50a2-48fb-9412-1b64cec4ec3b" containerName="authorino"
Jul 31 23:53:55.354893 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.354703    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a73afb8-5a54-4520-9f83-5cf0d24e434b" containerName="authorino"
Jul 31 23:53:55.354893 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.354734    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="76aec74b-50a2-48fb-9412-1b64cec4ec3b" containerName="authorino"
Jul 31 23:53:55.358027 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.358008    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.371373 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.371349    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"]
Jul 31 23:53:55.469564 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.469535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/92da091f-e08b-4bc5-865f-9b153946f58d-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.469755 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.469585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92da091f-e08b-4bc5-865f-9b153946f58d-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.469755 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.469623    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92da091f-e08b-4bc5-865f-9b153946f58d-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.469755 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.469641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/92da091f-e08b-4bc5-865f-9b153946f58d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.469755 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.469661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/92da091f-e08b-4bc5-865f-9b153946f58d-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.469755 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.469700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/92da091f-e08b-4bc5-865f-9b153946f58d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.469939 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.469761    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7w76\" (UniqueName: \"kubernetes.io/projected/92da091f-e08b-4bc5-865f-9b153946f58d-kube-api-access-k7w76\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.571017 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.570966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92da091f-e08b-4bc5-865f-9b153946f58d-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.571017 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.571015    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/92da091f-e08b-4bc5-865f-9b153946f58d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.571265 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.571046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/92da091f-e08b-4bc5-865f-9b153946f58d-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.571265 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.571067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/92da091f-e08b-4bc5-865f-9b153946f58d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.571265 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.571085    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7w76\" (UniqueName: \"kubernetes.io/projected/92da091f-e08b-4bc5-865f-9b153946f58d-kube-api-access-k7w76\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.571265 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.571156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/92da091f-e08b-4bc5-865f-9b153946f58d-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.571265 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.571223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92da091f-e08b-4bc5-865f-9b153946f58d-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.572096 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.572071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/92da091f-e08b-4bc5-865f-9b153946f58d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.573640 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.573618    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/92da091f-e08b-4bc5-865f-9b153946f58d-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.573764 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.573675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/92da091f-e08b-4bc5-865f-9b153946f58d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.574082 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.574060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92da091f-e08b-4bc5-865f-9b153946f58d-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.574123 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.574069    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/92da091f-e08b-4bc5-865f-9b153946f58d-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.581117 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.581095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92da091f-e08b-4bc5-865f-9b153946f58d-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.581479 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.581458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7w76\" (UniqueName: \"kubernetes.io/projected/92da091f-e08b-4bc5-865f-9b153946f58d-kube-api-access-k7w76\") pod \"istiod-openshift-gateway-5677ccb49b-8sb7w\" (UID: \"92da091f-e08b-4bc5-865f-9b153946f58d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.667396 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.667365    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:55.805762 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.805677    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"]
Jul 31 23:53:55.808074 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:53:55.808053    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92da091f_e08b_4bc5_865f_9b153946f58d.slice/crio-e9a13608d18b68b332866e434791b83e29de9f68dd56c6516ca01631950412de WatchSource:0}: Error finding container e9a13608d18b68b332866e434791b83e29de9f68dd56c6516ca01631950412de: Status 404 returned error can't find the container with id e9a13608d18b68b332866e434791b83e29de9f68dd56c6516ca01631950412de
Jul 31 23:53:55.810096 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.810068    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 31 23:53:55.810161 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:55.810125    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 31 23:53:56.487222 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:56.486564    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w" event={"ID":"92da091f-e08b-4bc5-865f-9b153946f58d","Type":"ContainerStarted","Data":"6eb6c7ee2636d5b73ed448b70a226d657210250a2454a2fe1af6cc08ea417d63"}
Jul 31 23:53:56.487222 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:56.486638    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w" event={"ID":"92da091f-e08b-4bc5-865f-9b153946f58d","Type":"ContainerStarted","Data":"e9a13608d18b68b332866e434791b83e29de9f68dd56c6516ca01631950412de"}
Jul 31 23:53:56.487222 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:56.486768    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:56.489895 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:56.489863    2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-8sb7w container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 31 23:53:56.490027 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:56.489915    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w" podUID="92da091f-e08b-4bc5-865f-9b153946f58d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:53:56.506771 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:56.506680    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w" podStartSLOduration=1.506660796 podStartE2EDuration="1.506660796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:53:56.505529686 +0000 UTC m=+475.865420769" watchObservedRunningTime="2026-07-31 23:53:56.506660796 +0000 UTC m=+475.866551836"
Jul 31 23:53:57.495158 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.495129    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8sb7w"
Jul 31 23:53:57.549391 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.549360    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"]
Jul 31 23:53:57.549681 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.549578    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g" podUID="9d964a45-cbe1-4201-a51f-797deac2acf4" containerName="discovery" containerID="cri-o://4859851566cff37c3ee2337f81c4b768fa513fb3d17024a666c1e8472458abda" gracePeriod=30
Jul 31 23:53:57.800800 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.800776    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:53:57.891008 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.890971    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndztb\" (UniqueName: \"kubernetes.io/projected/9d964a45-cbe1-4201-a51f-797deac2acf4-kube-api-access-ndztb\") pod \"9d964a45-cbe1-4201-a51f-797deac2acf4\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") "
Jul 31 23:53:57.891008 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.891009    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-csr-dns-cert\") pod \"9d964a45-cbe1-4201-a51f-797deac2acf4\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") "
Jul 31 23:53:57.891259 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.891039    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d964a45-cbe1-4201-a51f-797deac2acf4-local-certs\") pod \"9d964a45-cbe1-4201-a51f-797deac2acf4\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") "
Jul 31 23:53:57.891259 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.891061    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-token\") pod \"9d964a45-cbe1-4201-a51f-797deac2acf4\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") "
Jul 31 23:53:57.891259 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.891113    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-kubeconfig\") pod \"9d964a45-cbe1-4201-a51f-797deac2acf4\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") "
Jul 31 23:53:57.891259 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.891133    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9d964a45-cbe1-4201-a51f-797deac2acf4-cacerts\") pod \"9d964a45-cbe1-4201-a51f-797deac2acf4\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") "
Jul 31 23:53:57.891259 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.891163    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-csr-ca-configmap\") pod \"9d964a45-cbe1-4201-a51f-797deac2acf4\" (UID: \"9d964a45-cbe1-4201-a51f-797deac2acf4\") "
Jul 31 23:53:57.891787 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.891707    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "9d964a45-cbe1-4201-a51f-797deac2acf4" (UID: "9d964a45-cbe1-4201-a51f-797deac2acf4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:53:57.893773 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.893728    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "9d964a45-cbe1-4201-a51f-797deac2acf4" (UID: "9d964a45-cbe1-4201-a51f-797deac2acf4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:53:57.893773 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.893750    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d964a45-cbe1-4201-a51f-797deac2acf4-local-certs" (OuterVolumeSpecName: "local-certs") pod "9d964a45-cbe1-4201-a51f-797deac2acf4" (UID: "9d964a45-cbe1-4201-a51f-797deac2acf4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:53:57.893933 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.893795    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-token" (OuterVolumeSpecName: "istio-token") pod "9d964a45-cbe1-4201-a51f-797deac2acf4" (UID: "9d964a45-cbe1-4201-a51f-797deac2acf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:53:57.893933 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.893823    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "9d964a45-cbe1-4201-a51f-797deac2acf4" (UID: "9d964a45-cbe1-4201-a51f-797deac2acf4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:53:57.894005 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.893931    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d964a45-cbe1-4201-a51f-797deac2acf4-cacerts" (OuterVolumeSpecName: "cacerts") pod "9d964a45-cbe1-4201-a51f-797deac2acf4" (UID: "9d964a45-cbe1-4201-a51f-797deac2acf4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:53:57.894086 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.894071    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d964a45-cbe1-4201-a51f-797deac2acf4-kube-api-access-ndztb" (OuterVolumeSpecName: "kube-api-access-ndztb") pod "9d964a45-cbe1-4201-a51f-797deac2acf4" (UID: "9d964a45-cbe1-4201-a51f-797deac2acf4"). InnerVolumeSpecName "kube-api-access-ndztb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:53:57.992123 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.992090    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-token\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:53:57.992123 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.992113    2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-kubeconfig\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:53:57.992123 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.992123    2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9d964a45-cbe1-4201-a51f-797deac2acf4-cacerts\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:53:57.992123 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.992131    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-csr-ca-configmap\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:53:57.992371 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.992140    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndztb\" (UniqueName: \"kubernetes.io/projected/9d964a45-cbe1-4201-a51f-797deac2acf4-kube-api-access-ndztb\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:53:57.992371 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.992149    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9d964a45-cbe1-4201-a51f-797deac2acf4-istio-csr-dns-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:53:57.992371 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:57.992157    2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d964a45-cbe1-4201-a51f-797deac2acf4-local-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:53:58.500326 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:58.500294    2578 generic.go:356] "Generic (PLEG): container finished" podID="9d964a45-cbe1-4201-a51f-797deac2acf4" containerID="4859851566cff37c3ee2337f81c4b768fa513fb3d17024a666c1e8472458abda" exitCode=0
Jul 31 23:53:58.500831 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:58.500339    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"
Jul 31 23:53:58.500831 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:58.500373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g" event={"ID":"9d964a45-cbe1-4201-a51f-797deac2acf4","Type":"ContainerDied","Data":"4859851566cff37c3ee2337f81c4b768fa513fb3d17024a666c1e8472458abda"}
Jul 31 23:53:58.500831 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:58.500427    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g" event={"ID":"9d964a45-cbe1-4201-a51f-797deac2acf4","Type":"ContainerDied","Data":"af6f0c9c9534edf6fc1d7d20c0ff759b65b7e46d0518cb5df41a9f23de7cbce6"}
Jul 31 23:53:58.500831 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:58.500457    2578 scope.go:117] "RemoveContainer" containerID="4859851566cff37c3ee2337f81c4b768fa513fb3d17024a666c1e8472458abda"
Jul 31 23:53:58.510128 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:58.510113    2578 scope.go:117] "RemoveContainer" containerID="4859851566cff37c3ee2337f81c4b768fa513fb3d17024a666c1e8472458abda"
Jul 31 23:53:58.510360 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:53:58.510342    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4859851566cff37c3ee2337f81c4b768fa513fb3d17024a666c1e8472458abda\": container with ID starting with 4859851566cff37c3ee2337f81c4b768fa513fb3d17024a666c1e8472458abda not found: ID does not exist" containerID="4859851566cff37c3ee2337f81c4b768fa513fb3d17024a666c1e8472458abda"
Jul 31 23:53:58.510403 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:58.510369    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4859851566cff37c3ee2337f81c4b768fa513fb3d17024a666c1e8472458abda"} err="failed to get container status \"4859851566cff37c3ee2337f81c4b768fa513fb3d17024a666c1e8472458abda\": rpc error: code = NotFound desc = could not find container \"4859851566cff37c3ee2337f81c4b768fa513fb3d17024a666c1e8472458abda\": container with ID starting with 4859851566cff37c3ee2337f81c4b768fa513fb3d17024a666c1e8472458abda not found: ID does not exist"
Jul 31 23:53:58.533701 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:58.533675    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"]
Jul 31 23:53:58.540185 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:58.540165    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lz85g"]
Jul 31 23:53:59.187483 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:53:59.187452    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d964a45-cbe1-4201-a51f-797deac2acf4" path="/var/lib/kubelet/pods/9d964a45-cbe1-4201-a51f-797deac2acf4/volumes"
Jul 31 23:54:06.460788 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.460753    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-4mrls"]
Jul 31 23:54:06.461168 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.461114    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d964a45-cbe1-4201-a51f-797deac2acf4" containerName="discovery"
Jul 31 23:54:06.461168 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.461124    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d964a45-cbe1-4201-a51f-797deac2acf4" containerName="discovery"
Jul 31 23:54:06.461239 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.461175    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d964a45-cbe1-4201-a51f-797deac2acf4" containerName="discovery"
Jul 31 23:54:06.467149 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.467114    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-4mrls"
Jul 31 23:54:06.469444 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.469415    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 31 23:54:06.470360 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.470334    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 31 23:54:06.470491 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.470389    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 31 23:54:06.470491 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.470332    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-zt5mv\""
Jul 31 23:54:06.473364 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.473338    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-4mrls"]
Jul 31 23:54:06.504182 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.504152    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-tgv9k"]
Jul 31 23:54:06.507852 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.507829    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-tgv9k"
Jul 31 23:54:06.509825 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.509807    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 31 23:54:06.509934 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.509875    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-wdd7c\""
Jul 31 23:54:06.518231 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.518208    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-tgv9k"]
Jul 31 23:54:06.571423 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.571391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12e55187-b066-47d8-b15f-917341964598-cert\") pod \"kserve-controller-manager-674d959fb8-4mrls\" (UID: \"12e55187-b066-47d8-b15f-917341964598\") " pod="kserve/kserve-controller-manager-674d959fb8-4mrls"
Jul 31 23:54:06.571602 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.571433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95nq6\" (UniqueName: \"kubernetes.io/projected/12e55187-b066-47d8-b15f-917341964598-kube-api-access-95nq6\") pod \"kserve-controller-manager-674d959fb8-4mrls\" (UID: \"12e55187-b066-47d8-b15f-917341964598\") " pod="kserve/kserve-controller-manager-674d959fb8-4mrls"
Jul 31 23:54:06.672238 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.672207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/01882f1e-fae3-4c5c-a021-024b971ede1a-data\") pod \"seaweedfs-64568bcd49-tgv9k\" (UID: \"01882f1e-fae3-4c5c-a021-024b971ede1a\") " pod="kserve/seaweedfs-64568bcd49-tgv9k"
Jul 31 23:54:06.672425 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.672247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdwp4\" (UniqueName: \"kubernetes.io/projected/01882f1e-fae3-4c5c-a021-024b971ede1a-kube-api-access-zdwp4\") pod \"seaweedfs-64568bcd49-tgv9k\" (UID: \"01882f1e-fae3-4c5c-a021-024b971ede1a\") " pod="kserve/seaweedfs-64568bcd49-tgv9k"
Jul 31 23:54:06.672425 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.672297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12e55187-b066-47d8-b15f-917341964598-cert\") pod \"kserve-controller-manager-674d959fb8-4mrls\" (UID: \"12e55187-b066-47d8-b15f-917341964598\") " pod="kserve/kserve-controller-manager-674d959fb8-4mrls"
Jul 31 23:54:06.672425 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.672362    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95nq6\" (UniqueName: \"kubernetes.io/projected/12e55187-b066-47d8-b15f-917341964598-kube-api-access-95nq6\") pod \"kserve-controller-manager-674d959fb8-4mrls\" (UID: \"12e55187-b066-47d8-b15f-917341964598\") " pod="kserve/kserve-controller-manager-674d959fb8-4mrls"
Jul 31 23:54:06.672425 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:54:06.672411    2578 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 31 23:54:06.672559 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:54:06.672488    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e55187-b066-47d8-b15f-917341964598-cert podName:12e55187-b066-47d8-b15f-917341964598 nodeName:}" failed. No retries permitted until 2026-07-31 23:54:07.172471282 +0000 UTC m=+486.532362302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/12e55187-b066-47d8-b15f-917341964598-cert") pod "kserve-controller-manager-674d959fb8-4mrls" (UID: "12e55187-b066-47d8-b15f-917341964598") : secret "kserve-webhook-server-cert" not found
Jul 31 23:54:06.682340 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.682308    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95nq6\" (UniqueName: \"kubernetes.io/projected/12e55187-b066-47d8-b15f-917341964598-kube-api-access-95nq6\") pod \"kserve-controller-manager-674d959fb8-4mrls\" (UID: \"12e55187-b066-47d8-b15f-917341964598\") " pod="kserve/kserve-controller-manager-674d959fb8-4mrls"
Jul 31 23:54:06.772941 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.772840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/01882f1e-fae3-4c5c-a021-024b971ede1a-data\") pod \"seaweedfs-64568bcd49-tgv9k\" (UID: \"01882f1e-fae3-4c5c-a021-024b971ede1a\") " pod="kserve/seaweedfs-64568bcd49-tgv9k"
Jul 31 23:54:06.772941 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.772885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdwp4\" (UniqueName: \"kubernetes.io/projected/01882f1e-fae3-4c5c-a021-024b971ede1a-kube-api-access-zdwp4\") pod \"seaweedfs-64568bcd49-tgv9k\" (UID: \"01882f1e-fae3-4c5c-a021-024b971ede1a\") " pod="kserve/seaweedfs-64568bcd49-tgv9k"
Jul 31 23:54:06.773231 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.773212    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/01882f1e-fae3-4c5c-a021-024b971ede1a-data\") pod \"seaweedfs-64568bcd49-tgv9k\" (UID: \"01882f1e-fae3-4c5c-a021-024b971ede1a\") " pod="kserve/seaweedfs-64568bcd49-tgv9k"
Jul 31 23:54:06.781147 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.781118    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdwp4\" (UniqueName: \"kubernetes.io/projected/01882f1e-fae3-4c5c-a021-024b971ede1a-kube-api-access-zdwp4\") pod \"seaweedfs-64568bcd49-tgv9k\" (UID: \"01882f1e-fae3-4c5c-a021-024b971ede1a\") " pod="kserve/seaweedfs-64568bcd49-tgv9k"
Jul 31 23:54:06.817757 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.817698    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-tgv9k"
Jul 31 23:54:06.966740 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:06.966699    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-tgv9k"]
Jul 31 23:54:06.968206 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:54:06.968181    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01882f1e_fae3_4c5c_a021_024b971ede1a.slice/crio-0f1a3d4257d0494e2c50ae71285e4970747a2f0267dc92fd8e7a921334eb3303 WatchSource:0}: Error finding container 0f1a3d4257d0494e2c50ae71285e4970747a2f0267dc92fd8e7a921334eb3303: Status 404 returned error can't find the container with id 0f1a3d4257d0494e2c50ae71285e4970747a2f0267dc92fd8e7a921334eb3303
Jul 31 23:54:07.178202 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:07.178164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12e55187-b066-47d8-b15f-917341964598-cert\") pod \"kserve-controller-manager-674d959fb8-4mrls\" (UID: \"12e55187-b066-47d8-b15f-917341964598\") " pod="kserve/kserve-controller-manager-674d959fb8-4mrls"
Jul 31 23:54:07.181214 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:07.181191    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12e55187-b066-47d8-b15f-917341964598-cert\") pod \"kserve-controller-manager-674d959fb8-4mrls\" (UID: \"12e55187-b066-47d8-b15f-917341964598\") " pod="kserve/kserve-controller-manager-674d959fb8-4mrls"
Jul 31 23:54:07.379825 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:07.379785    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-4mrls"
Jul 31 23:54:07.537604 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:07.537580    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-4mrls"]
Jul 31 23:54:07.554334 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:07.554303    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-tgv9k" event={"ID":"01882f1e-fae3-4c5c-a021-024b971ede1a","Type":"ContainerStarted","Data":"0f1a3d4257d0494e2c50ae71285e4970747a2f0267dc92fd8e7a921334eb3303"}
Jul 31 23:54:07.652088 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:54:07.652050    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e55187_b066_47d8_b15f_917341964598.slice/crio-f8cc11b37cb2c94c42d2f2479fd9719c3f9732735aaea6fd4fe7cf701724f36b WatchSource:0}: Error finding container f8cc11b37cb2c94c42d2f2479fd9719c3f9732735aaea6fd4fe7cf701724f36b: Status 404 returned error can't find the container with id f8cc11b37cb2c94c42d2f2479fd9719c3f9732735aaea6fd4fe7cf701724f36b
Jul 31 23:54:08.565354 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:08.565276    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-4mrls" event={"ID":"12e55187-b066-47d8-b15f-917341964598","Type":"ContainerStarted","Data":"f8cc11b37cb2c94c42d2f2479fd9719c3f9732735aaea6fd4fe7cf701724f36b"}
Jul 31 23:54:11.592345 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:11.592302    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-tgv9k" event={"ID":"01882f1e-fae3-4c5c-a021-024b971ede1a","Type":"ContainerStarted","Data":"5216e823ccf8f96f1f5a81ff3e1a7f021fd65cd94c9fb8436f46593dc7707424"}
Jul 31 23:54:11.592853 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:11.592395    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-tgv9k"
Jul 31 23:54:11.593847 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:11.593824    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-4mrls" event={"ID":"12e55187-b066-47d8-b15f-917341964598","Type":"ContainerStarted","Data":"d94513c2906326bd778c0fbea7e9f0ab328e06c8225cd1c44368312864e53d81"}
Jul 31 23:54:11.593976 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:11.593934    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-674d959fb8-4mrls"
Jul 31 23:54:11.609381 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:11.609301    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-tgv9k" podStartSLOduration=1.198310898 podStartE2EDuration="5.609286243s" totalImagesPullingTime="4.410975345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:54:11.606902152 +0000 UTC m=+490.966793234" watchObservedRunningTime="2026-07-31 23:54:11.609286243 +0000 UTC m=+490.969177283"
Jul 31 23:54:11.622270 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:11.622212    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-674d959fb8-4mrls" podStartSLOduration=1.9914236829999998 podStartE2EDuration="5.622198512s" totalImagesPullingTime="3.630774829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:54:11.620067559 +0000 UTC m=+490.979958620" watchObservedRunningTime="2026-07-31 23:54:11.622198512 +0000 UTC m=+490.982089553"
Jul 31 23:54:17.601237 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:17.601204    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-tgv9k"
Jul 31 23:54:42.604942 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:42.604906    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-674d959fb8-4mrls"
Jul 31 23:54:49.296567 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.296532    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-4mrls"]
Jul 31 23:54:49.297088 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.296802    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-674d959fb8-4mrls" podUID="12e55187-b066-47d8-b15f-917341964598" containerName="manager" containerID="cri-o://d94513c2906326bd778c0fbea7e9f0ab328e06c8225cd1c44368312864e53d81" gracePeriod=10
Jul 31 23:54:49.322787 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.322757    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-zbm2g"]
Jul 31 23:54:49.326673 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.326654    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-zbm2g"
Jul 31 23:54:49.333084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.333060    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-zbm2g"]
Jul 31 23:54:49.449988 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.449952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8zqx\" (UniqueName: \"kubernetes.io/projected/5dbb392c-4aab-4c3e-ac0e-7f578ff3a398-kube-api-access-x8zqx\") pod \"kserve-controller-manager-674d959fb8-zbm2g\" (UID: \"5dbb392c-4aab-4c3e-ac0e-7f578ff3a398\") " pod="kserve/kserve-controller-manager-674d959fb8-zbm2g"
Jul 31 23:54:49.450155 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.450027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5dbb392c-4aab-4c3e-ac0e-7f578ff3a398-cert\") pod \"kserve-controller-manager-674d959fb8-zbm2g\" (UID: \"5dbb392c-4aab-4c3e-ac0e-7f578ff3a398\") " pod="kserve/kserve-controller-manager-674d959fb8-zbm2g"
Jul 31 23:54:49.550853 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.550823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5dbb392c-4aab-4c3e-ac0e-7f578ff3a398-cert\") pod \"kserve-controller-manager-674d959fb8-zbm2g\" (UID: \"5dbb392c-4aab-4c3e-ac0e-7f578ff3a398\") " pod="kserve/kserve-controller-manager-674d959fb8-zbm2g"
Jul 31 23:54:49.551030 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.550903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8zqx\" (UniqueName: \"kubernetes.io/projected/5dbb392c-4aab-4c3e-ac0e-7f578ff3a398-kube-api-access-x8zqx\") pod \"kserve-controller-manager-674d959fb8-zbm2g\" (UID: \"5dbb392c-4aab-4c3e-ac0e-7f578ff3a398\") " pod="kserve/kserve-controller-manager-674d959fb8-zbm2g"
Jul 31 23:54:49.553371 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.553348    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5dbb392c-4aab-4c3e-ac0e-7f578ff3a398-cert\") pod \"kserve-controller-manager-674d959fb8-zbm2g\" (UID: \"5dbb392c-4aab-4c3e-ac0e-7f578ff3a398\") " pod="kserve/kserve-controller-manager-674d959fb8-zbm2g"
Jul 31 23:54:49.559094 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.559062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8zqx\" (UniqueName: \"kubernetes.io/projected/5dbb392c-4aab-4c3e-ac0e-7f578ff3a398-kube-api-access-x8zqx\") pod \"kserve-controller-manager-674d959fb8-zbm2g\" (UID: \"5dbb392c-4aab-4c3e-ac0e-7f578ff3a398\") " pod="kserve/kserve-controller-manager-674d959fb8-zbm2g"
Jul 31 23:54:49.577645 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.577616    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-4mrls"
Jul 31 23:54:49.651820 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.651773    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12e55187-b066-47d8-b15f-917341964598-cert\") pod \"12e55187-b066-47d8-b15f-917341964598\" (UID: \"12e55187-b066-47d8-b15f-917341964598\") "
Jul 31 23:54:49.652025 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.651868    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95nq6\" (UniqueName: \"kubernetes.io/projected/12e55187-b066-47d8-b15f-917341964598-kube-api-access-95nq6\") pod \"12e55187-b066-47d8-b15f-917341964598\" (UID: \"12e55187-b066-47d8-b15f-917341964598\") "
Jul 31 23:54:49.654034 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.654008    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e55187-b066-47d8-b15f-917341964598-cert" (OuterVolumeSpecName: "cert") pod "12e55187-b066-47d8-b15f-917341964598" (UID: "12e55187-b066-47d8-b15f-917341964598"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:54:49.654130 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.654041    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e55187-b066-47d8-b15f-917341964598-kube-api-access-95nq6" (OuterVolumeSpecName: "kube-api-access-95nq6") pod "12e55187-b066-47d8-b15f-917341964598" (UID: "12e55187-b066-47d8-b15f-917341964598"). InnerVolumeSpecName "kube-api-access-95nq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:54:49.693386 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.693353    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-zbm2g"
Jul 31 23:54:49.752902 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.752866    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95nq6\" (UniqueName: \"kubernetes.io/projected/12e55187-b066-47d8-b15f-917341964598-kube-api-access-95nq6\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:54:49.753062 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.752906    2578 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12e55187-b066-47d8-b15f-917341964598-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:54:49.797957 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.797633    2578 generic.go:356] "Generic (PLEG): container finished" podID="12e55187-b066-47d8-b15f-917341964598" containerID="d94513c2906326bd778c0fbea7e9f0ab328e06c8225cd1c44368312864e53d81" exitCode=0
Jul 31 23:54:49.797957 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.797684    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-4mrls"
Jul 31 23:54:49.797957 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.797806    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-4mrls" event={"ID":"12e55187-b066-47d8-b15f-917341964598","Type":"ContainerDied","Data":"d94513c2906326bd778c0fbea7e9f0ab328e06c8225cd1c44368312864e53d81"}
Jul 31 23:54:49.797957 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.797875    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-4mrls" event={"ID":"12e55187-b066-47d8-b15f-917341964598","Type":"ContainerDied","Data":"f8cc11b37cb2c94c42d2f2479fd9719c3f9732735aaea6fd4fe7cf701724f36b"}
Jul 31 23:54:49.797957 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.797902    2578 scope.go:117] "RemoveContainer" containerID="d94513c2906326bd778c0fbea7e9f0ab328e06c8225cd1c44368312864e53d81"
Jul 31 23:54:49.809053 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.809034    2578 scope.go:117] "RemoveContainer" containerID="d94513c2906326bd778c0fbea7e9f0ab328e06c8225cd1c44368312864e53d81"
Jul 31 23:54:49.809380 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:54:49.809359    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94513c2906326bd778c0fbea7e9f0ab328e06c8225cd1c44368312864e53d81\": container with ID starting with d94513c2906326bd778c0fbea7e9f0ab328e06c8225cd1c44368312864e53d81 not found: ID does not exist" containerID="d94513c2906326bd778c0fbea7e9f0ab328e06c8225cd1c44368312864e53d81"
Jul 31 23:54:49.809458 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.809393    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94513c2906326bd778c0fbea7e9f0ab328e06c8225cd1c44368312864e53d81"} err="failed to get container status \"d94513c2906326bd778c0fbea7e9f0ab328e06c8225cd1c44368312864e53d81\": rpc error: code = NotFound desc = could not find container \"d94513c2906326bd778c0fbea7e9f0ab328e06c8225cd1c44368312864e53d81\": container with ID starting with d94513c2906326bd778c0fbea7e9f0ab328e06c8225cd1c44368312864e53d81 not found: ID does not exist"
Jul 31 23:54:49.819154 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.819127    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-4mrls"]
Jul 31 23:54:49.826030 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.826008    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-4mrls"]
Jul 31 23:54:49.830728 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:49.830690    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-zbm2g"]
Jul 31 23:54:49.834406 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:54:49.834376    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dbb392c_4aab_4c3e_ac0e_7f578ff3a398.slice/crio-983cb8e88ffac0fb4835ec39b6ba0baa2a27eb7a9a357035d7dffb086c07f907 WatchSource:0}: Error finding container 983cb8e88ffac0fb4835ec39b6ba0baa2a27eb7a9a357035d7dffb086c07f907: Status 404 returned error can't find the container with id 983cb8e88ffac0fb4835ec39b6ba0baa2a27eb7a9a357035d7dffb086c07f907
Jul 31 23:54:50.804430 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:50.804395    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-zbm2g" event={"ID":"5dbb392c-4aab-4c3e-ac0e-7f578ff3a398","Type":"ContainerStarted","Data":"95b55fb09a183eb3c12954419360ad9de9514ee60db75a42dc64fab12b2d4919"}
Jul 31 23:54:50.804901 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:50.804454    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-zbm2g" event={"ID":"5dbb392c-4aab-4c3e-ac0e-7f578ff3a398","Type":"ContainerStarted","Data":"983cb8e88ffac0fb4835ec39b6ba0baa2a27eb7a9a357035d7dffb086c07f907"}
Jul 31 23:54:50.804901 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:50.804496    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-674d959fb8-zbm2g"
Jul 31 23:54:50.822301 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:50.822251    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-674d959fb8-zbm2g" podStartSLOduration=1.337313846 podStartE2EDuration="1.822235925s" totalImagesPullingTime="484.922079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:54:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:54:50.819850753 +0000 UTC m=+530.179741810" watchObservedRunningTime="2026-07-31 23:54:50.822235925 +0000 UTC m=+530.182126966"
Jul 31 23:54:51.186074 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:54:51.186041    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12e55187-b066-47d8-b15f-917341964598" path="/var/lib/kubelet/pods/12e55187-b066-47d8-b15f-917341964598/volumes"
Jul 31 23:55:21.815653 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:21.815621    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-674d959fb8-zbm2g"
Jul 31 23:55:22.846994 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.846963    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-95fqk"]
Jul 31 23:55:22.847368 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.847356    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12e55187-b066-47d8-b15f-917341964598" containerName="manager"
Jul 31 23:55:22.847368 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.847367    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="12e55187-b066-47d8-b15f-917341964598" containerName="manager"
Jul 31 23:55:22.847437 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.847426    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="12e55187-b066-47d8-b15f-917341964598" containerName="manager"
Jul 31 23:55:22.850479 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.850450    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-95fqk"
Jul 31 23:55:22.852467 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.852448    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 31 23:55:22.852467 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.852458    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-dww62\""
Jul 31 23:55:22.858380 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.858359    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-95fqk"]
Jul 31 23:55:22.861569 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.861545    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-8588677cfb-srd4w"]
Jul 31 23:55:22.865608 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.865585    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-srd4w"
Jul 31 23:55:22.868351 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.868331    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-q28ht\""
Jul 31 23:55:22.868527 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.868331    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Jul 31 23:55:22.868642 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.868394    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 31 23:55:22.875696 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.875669    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-srd4w"]
Jul 31 23:55:22.945196 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.945164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6aed191-891b-4fb8-ae0c-96e786b71747-metrics-certs\") pod \"odh-model-controller-8588677cfb-srd4w\" (UID: \"c6aed191-891b-4fb8-ae0c-96e786b71747\") " pod="kserve/odh-model-controller-8588677cfb-srd4w"
Jul 31 23:55:22.945381 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.945202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2ldw\" (UniqueName: \"kubernetes.io/projected/c6aed191-891b-4fb8-ae0c-96e786b71747-kube-api-access-x2ldw\") pod \"odh-model-controller-8588677cfb-srd4w\" (UID: \"c6aed191-891b-4fb8-ae0c-96e786b71747\") " pod="kserve/odh-model-controller-8588677cfb-srd4w"
Jul 31 23:55:22.945381 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.945231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72dc53a4-36dc-4542-9571-6c7ecd502cae-tls-certs\") pod \"model-serving-api-7b954ffc95-95fqk\" (UID: \"72dc53a4-36dc-4542-9571-6c7ecd502cae\") " pod="kserve/model-serving-api-7b954ffc95-95fqk"
Jul 31 23:55:22.945381 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.945249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq5zq\" (UniqueName: \"kubernetes.io/projected/72dc53a4-36dc-4542-9571-6c7ecd502cae-kube-api-access-cq5zq\") pod \"model-serving-api-7b954ffc95-95fqk\" (UID: \"72dc53a4-36dc-4542-9571-6c7ecd502cae\") " pod="kserve/model-serving-api-7b954ffc95-95fqk"
Jul 31 23:55:22.945381 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:22.945330    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6aed191-891b-4fb8-ae0c-96e786b71747-cert\") pod \"odh-model-controller-8588677cfb-srd4w\" (UID: \"c6aed191-891b-4fb8-ae0c-96e786b71747\") " pod="kserve/odh-model-controller-8588677cfb-srd4w"
Jul 31 23:55:23.046375 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:23.046342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6aed191-891b-4fb8-ae0c-96e786b71747-cert\") pod \"odh-model-controller-8588677cfb-srd4w\" (UID: \"c6aed191-891b-4fb8-ae0c-96e786b71747\") " pod="kserve/odh-model-controller-8588677cfb-srd4w"
Jul 31 23:55:23.046517 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:23.046406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6aed191-891b-4fb8-ae0c-96e786b71747-metrics-certs\") pod \"odh-model-controller-8588677cfb-srd4w\" (UID: \"c6aed191-891b-4fb8-ae0c-96e786b71747\") " pod="kserve/odh-model-controller-8588677cfb-srd4w"
Jul 31 23:55:23.046517 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:23.046424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2ldw\" (UniqueName: \"kubernetes.io/projected/c6aed191-891b-4fb8-ae0c-96e786b71747-kube-api-access-x2ldw\") pod \"odh-model-controller-8588677cfb-srd4w\" (UID: \"c6aed191-891b-4fb8-ae0c-96e786b71747\") " pod="kserve/odh-model-controller-8588677cfb-srd4w"
Jul 31 23:55:23.046517 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:23.046447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72dc53a4-36dc-4542-9571-6c7ecd502cae-tls-certs\") pod \"model-serving-api-7b954ffc95-95fqk\" (UID: \"72dc53a4-36dc-4542-9571-6c7ecd502cae\") " pod="kserve/model-serving-api-7b954ffc95-95fqk"
Jul 31 23:55:23.046517 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:23.046466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq5zq\" (UniqueName: \"kubernetes.io/projected/72dc53a4-36dc-4542-9571-6c7ecd502cae-kube-api-access-cq5zq\") pod \"model-serving-api-7b954ffc95-95fqk\" (UID: \"72dc53a4-36dc-4542-9571-6c7ecd502cae\") " pod="kserve/model-serving-api-7b954ffc95-95fqk"
Jul 31 23:55:23.046765 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:55:23.046520    2578 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 31 23:55:23.046765 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:55:23.046585    2578 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 31 23:55:23.046765 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:55:23.046606    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6aed191-891b-4fb8-ae0c-96e786b71747-cert podName:c6aed191-891b-4fb8-ae0c-96e786b71747 nodeName:}" failed. No retries permitted until 2026-07-31 23:55:23.546583751 +0000 UTC m=+562.906474788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c6aed191-891b-4fb8-ae0c-96e786b71747-cert") pod "odh-model-controller-8588677cfb-srd4w" (UID: "c6aed191-891b-4fb8-ae0c-96e786b71747") : secret "odh-model-controller-webhook-cert" not found
Jul 31 23:55:23.046765 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:55:23.046645    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72dc53a4-36dc-4542-9571-6c7ecd502cae-tls-certs podName:72dc53a4-36dc-4542-9571-6c7ecd502cae nodeName:}" failed. No retries permitted until 2026-07-31 23:55:23.546627956 +0000 UTC m=+562.906518993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/72dc53a4-36dc-4542-9571-6c7ecd502cae-tls-certs") pod "model-serving-api-7b954ffc95-95fqk" (UID: "72dc53a4-36dc-4542-9571-6c7ecd502cae") : secret "model-serving-api-tls" not found
Jul 31 23:55:23.049078 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:23.049058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6aed191-891b-4fb8-ae0c-96e786b71747-metrics-certs\") pod \"odh-model-controller-8588677cfb-srd4w\" (UID: \"c6aed191-891b-4fb8-ae0c-96e786b71747\") " pod="kserve/odh-model-controller-8588677cfb-srd4w"
Jul 31 23:55:23.055808 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:23.055783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq5zq\" (UniqueName: \"kubernetes.io/projected/72dc53a4-36dc-4542-9571-6c7ecd502cae-kube-api-access-cq5zq\") pod \"model-serving-api-7b954ffc95-95fqk\" (UID: \"72dc53a4-36dc-4542-9571-6c7ecd502cae\") " pod="kserve/model-serving-api-7b954ffc95-95fqk"
Jul 31 23:55:23.056405 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:23.056385    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2ldw\" (UniqueName: \"kubernetes.io/projected/c6aed191-891b-4fb8-ae0c-96e786b71747-kube-api-access-x2ldw\") pod \"odh-model-controller-8588677cfb-srd4w\" (UID: \"c6aed191-891b-4fb8-ae0c-96e786b71747\") " pod="kserve/odh-model-controller-8588677cfb-srd4w"
Jul 31 23:55:23.552401 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:23.552373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72dc53a4-36dc-4542-9571-6c7ecd502cae-tls-certs\") pod \"model-serving-api-7b954ffc95-95fqk\" (UID: \"72dc53a4-36dc-4542-9571-6c7ecd502cae\") " pod="kserve/model-serving-api-7b954ffc95-95fqk"
Jul 31 23:55:23.552580 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:23.552416    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6aed191-891b-4fb8-ae0c-96e786b71747-cert\") pod \"odh-model-controller-8588677cfb-srd4w\" (UID: \"c6aed191-891b-4fb8-ae0c-96e786b71747\") " pod="kserve/odh-model-controller-8588677cfb-srd4w"
Jul 31 23:55:23.554906 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:23.554874    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72dc53a4-36dc-4542-9571-6c7ecd502cae-tls-certs\") pod \"model-serving-api-7b954ffc95-95fqk\" (UID: \"72dc53a4-36dc-4542-9571-6c7ecd502cae\") " pod="kserve/model-serving-api-7b954ffc95-95fqk"
Jul 31 23:55:23.555046 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:23.554928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6aed191-891b-4fb8-ae0c-96e786b71747-cert\") pod \"odh-model-controller-8588677cfb-srd4w\" (UID: \"c6aed191-891b-4fb8-ae0c-96e786b71747\") " pod="kserve/odh-model-controller-8588677cfb-srd4w"
Jul 31 23:55:23.762954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:23.762910    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-95fqk"
Jul 31 23:55:23.779248 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:23.779211    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-srd4w"
Jul 31 23:55:23.916560 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:23.916535    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-95fqk"]
Jul 31 23:55:23.918743 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:55:23.918693    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72dc53a4_36dc_4542_9571_6c7ecd502cae.slice/crio-28767a4ce3981d050f50998598681f404f496ec1ae4e9b6142316248c38009fb WatchSource:0}: Error finding container 28767a4ce3981d050f50998598681f404f496ec1ae4e9b6142316248c38009fb: Status 404 returned error can't find the container with id 28767a4ce3981d050f50998598681f404f496ec1ae4e9b6142316248c38009fb
Jul 31 23:55:23.937120 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:23.936984    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-srd4w"]
Jul 31 23:55:23.939364 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:55:23.939340    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6aed191_891b_4fb8_ae0c_96e786b71747.slice/crio-b60d3a4a82472fbff2debe6f4cab13a42af4e61aaa462da33d4fcc183d20d48d WatchSource:0}: Error finding container b60d3a4a82472fbff2debe6f4cab13a42af4e61aaa462da33d4fcc183d20d48d: Status 404 returned error can't find the container with id b60d3a4a82472fbff2debe6f4cab13a42af4e61aaa462da33d4fcc183d20d48d
Jul 31 23:55:23.991604 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:23.991577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-95fqk" event={"ID":"72dc53a4-36dc-4542-9571-6c7ecd502cae","Type":"ContainerStarted","Data":"28767a4ce3981d050f50998598681f404f496ec1ae4e9b6142316248c38009fb"}
Jul 31 23:55:23.992751 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:23.992729    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-srd4w" event={"ID":"c6aed191-891b-4fb8-ae0c-96e786b71747","Type":"ContainerStarted","Data":"b60d3a4a82472fbff2debe6f4cab13a42af4e61aaa462da33d4fcc183d20d48d"}
Jul 31 23:55:27.024304 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:27.024181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-srd4w" event={"ID":"c6aed191-891b-4fb8-ae0c-96e786b71747","Type":"ContainerStarted","Data":"e043ca8eb3956798b95824614465afa1bf5acfb9b85ef473c65fde75df8fd58f"}
Jul 31 23:55:27.024304 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:27.024296    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-8588677cfb-srd4w"
Jul 31 23:55:27.026106 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:27.026083    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-95fqk" event={"ID":"72dc53a4-36dc-4542-9571-6c7ecd502cae","Type":"ContainerStarted","Data":"8277476ef9ebe54a8dd795c0121a1bc57c2a15df54c37de3dd0ae332cdfb89ec"}
Jul 31 23:55:27.026214 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:27.026165    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-95fqk"
Jul 31 23:55:27.040608 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:27.040554    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-8588677cfb-srd4w" podStartSLOduration=2.265839973 podStartE2EDuration="5.040538014s" totalImagesPullingTime="2.774698041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:55:27.038981439 +0000 UTC m=+566.398872483" watchObservedRunningTime="2026-07-31 23:55:27.040538014 +0000 UTC m=+566.400429055"
Jul 31 23:55:27.056198 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:27.056148    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-95fqk" podStartSLOduration=2.313320037 podStartE2EDuration="5.05613316s" totalImagesPullingTime="2.742813123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:55:27.054367068 +0000 UTC m=+566.414258130" watchObservedRunningTime="2026-07-31 23:55:27.05613316 +0000 UTC m=+566.416024202"
Jul 31 23:55:38.034084 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:38.034053    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-8588677cfb-srd4w"
Jul 31 23:55:38.035871 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:55:38.035847    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-95fqk"
Jul 31 23:56:01.119538 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:56:01.119507    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Jul 31 23:56:01.121092 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:56:01.121074    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Jul 31 23:57:14.652788 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:14.652753    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6"]
Jul 31 23:57:14.656954 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:14.656934    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6"
Jul 31 23:57:14.660476 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:14.660455    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-sv6zn\""
Jul 31 23:57:14.660619 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:14.660461    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 31 23:57:14.660619 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:14.660508    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-dk5dz\""
Jul 31 23:57:14.661193 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:14.661176    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 31 23:57:14.663706 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:14.663446    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 31 23:57:14.691861 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:14.691835    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6"]
Jul 31 23:57:14.740668 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:14.740634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6862a8-2da5-494c-9e5e-0a0da4ce9acf-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6\" (UID: \"8e6862a8-2da5-494c-9e5e-0a0da4ce9acf\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6"
Jul 31 23:57:14.740888 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:14.740773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ndn9\" (UniqueName: \"kubernetes.io/projected/8e6862a8-2da5-494c-9e5e-0a0da4ce9acf-kube-api-access-4ndn9\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6\" (UID: \"8e6862a8-2da5-494c-9e5e-0a0da4ce9acf\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6"
Jul 31 23:57:14.841455 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:14.841420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ndn9\" (UniqueName: \"kubernetes.io/projected/8e6862a8-2da5-494c-9e5e-0a0da4ce9acf-kube-api-access-4ndn9\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6\" (UID: \"8e6862a8-2da5-494c-9e5e-0a0da4ce9acf\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6"
Jul 31 23:57:14.841623 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:14.841491    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6862a8-2da5-494c-9e5e-0a0da4ce9acf-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6\" (UID: \"8e6862a8-2da5-494c-9e5e-0a0da4ce9acf\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6"
Jul 31 23:57:14.844099 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:14.844065    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6862a8-2da5-494c-9e5e-0a0da4ce9acf-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6\" (UID: \"8e6862a8-2da5-494c-9e5e-0a0da4ce9acf\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6"
Jul 31 23:57:14.850258 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:14.850212    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ndn9\" (UniqueName: \"kubernetes.io/projected/8e6862a8-2da5-494c-9e5e-0a0da4ce9acf-kube-api-access-4ndn9\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6\" (UID: \"8e6862a8-2da5-494c-9e5e-0a0da4ce9acf\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6"
Jul 31 23:57:14.966969 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:14.966869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6"
Jul 31 23:57:15.108918 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:15.108739    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6"]
Jul 31 23:57:15.111275 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:57:15.111247    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6862a8_2da5_494c_9e5e_0a0da4ce9acf.slice/crio-7b7020144283a02fe817803362290f544be66425d78597a286dea5a648e1d778 WatchSource:0}: Error finding container 7b7020144283a02fe817803362290f544be66425d78597a286dea5a648e1d778: Status 404 returned error can't find the container with id 7b7020144283a02fe817803362290f544be66425d78597a286dea5a648e1d778
Jul 31 23:57:15.113405 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:15.113389    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 23:57:15.614089 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:15.614047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6" event={"ID":"8e6862a8-2da5-494c-9e5e-0a0da4ce9acf","Type":"ContainerStarted","Data":"7b7020144283a02fe817803362290f544be66425d78597a286dea5a648e1d778"}
Jul 31 23:57:18.634780 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:18.634738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6" event={"ID":"8e6862a8-2da5-494c-9e5e-0a0da4ce9acf","Type":"ContainerStarted","Data":"ee52536672307fe12174790705c64f188666e1fbe992077616f4655a0a0ac18d"}
Jul 31 23:57:18.635167 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:18.634867    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6"
Jul 31 23:57:18.649632 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:18.649578    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6" podStartSLOduration=2.183727232 podStartE2EDuration="4.649561258s" totalImagesPullingTime="2.465834026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:57:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:57:18.649302945 +0000 UTC m=+678.009193986" watchObservedRunningTime="2026-07-31 23:57:18.649561258 +0000 UTC m=+678.009452299"
Jul 31 23:57:30.081070 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.081031    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"]
Jul 31 23:57:30.084820 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.084797    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.086860 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.086837    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 31 23:57:30.086985 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.086858    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 31 23:57:30.086985 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.086883    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 31 23:57:30.086985 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.086837    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-ff85b\""
Jul 31 23:57:30.100396 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.100367    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"]
Jul 31 23:57:30.181479 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.181447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-zxhgq\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.181678 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.181488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-home\") pod \"fc-smoke-test-kserve-7f4687897-zxhgq\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.181678 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.181505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-dshm\") pod \"fc-smoke-test-kserve-7f4687897-zxhgq\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.181678 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.181522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6212069-4342-485f-9c35-edfe74d025ea-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-zxhgq\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.181678 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.181597    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8799m\" (UniqueName: \"kubernetes.io/projected/b6212069-4342-485f-9c35-edfe74d025ea-kube-api-access-8799m\") pod \"fc-smoke-test-kserve-7f4687897-zxhgq\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.181678 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.181673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-zxhgq\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.282842 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.282805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-zxhgq\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.283014 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.282900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-zxhgq\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.283014 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.282940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-home\") pod \"fc-smoke-test-kserve-7f4687897-zxhgq\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.283014 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.282967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-dshm\") pod \"fc-smoke-test-kserve-7f4687897-zxhgq\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.283014 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.282991    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6212069-4342-485f-9c35-edfe74d025ea-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-zxhgq\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.283239 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.283066    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8799m\" (UniqueName: \"kubernetes.io/projected/b6212069-4342-485f-9c35-edfe74d025ea-kube-api-access-8799m\") pod \"fc-smoke-test-kserve-7f4687897-zxhgq\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.283239 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.283200    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-zxhgq\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.283350 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.283258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-zxhgq\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.283586 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.283564    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-home\") pod \"fc-smoke-test-kserve-7f4687897-zxhgq\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.286204 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.286178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-dshm\") pod \"fc-smoke-test-kserve-7f4687897-zxhgq\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.286405 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.286389    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6212069-4342-485f-9c35-edfe74d025ea-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-zxhgq\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.295887 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.295805    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8799m\" (UniqueName: \"kubernetes.io/projected/b6212069-4342-485f-9c35-edfe74d025ea-kube-api-access-8799m\") pod \"fc-smoke-test-kserve-7f4687897-zxhgq\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.403640 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.403604    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:30.545119 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.545092    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"]
Jul 31 23:57:30.547060 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:57:30.547030    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6212069_4342_485f_9c35_edfe74d025ea.slice/crio-20442cef99de494f7f0007b168123dc8b9e4e7c0500c6cf38f9f6b6969736e61 WatchSource:0}: Error finding container 20442cef99de494f7f0007b168123dc8b9e4e7c0500c6cf38f9f6b6969736e61: Status 404 returned error can't find the container with id 20442cef99de494f7f0007b168123dc8b9e4e7c0500c6cf38f9f6b6969736e61
Jul 31 23:57:30.699430 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:30.699345    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq" event={"ID":"b6212069-4342-485f-9c35-edfe74d025ea","Type":"ContainerStarted","Data":"20442cef99de494f7f0007b168123dc8b9e4e7c0500c6cf38f9f6b6969736e61"}
Jul 31 23:57:32.714849 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:32.714816    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq" event={"ID":"b6212069-4342-485f-9c35-edfe74d025ea","Type":"ContainerStarted","Data":"f443fb9c48779c2ef46e5d18a2efe4f8bc3e5c6fa5f062eaac0d426ec9dc5e32"}
Jul 31 23:57:32.731368 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:32.731309    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq" podStartSLOduration=1.138125182 podStartE2EDuration="2.731285713s" totalImagesPullingTime="1.593160531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:57:32.730393151 +0000 UTC m=+692.090284192" watchObservedRunningTime="2026-07-31 23:57:32.731285713 +0000 UTC m=+692.091176755"
Jul 31 23:57:33.404308 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:33.404275    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:40.404222 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:40.404173    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:40.417047 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:40.417013    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:40.769577 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:40.769489    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:57:49.642457 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:57:49.642427    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6"
Jul 31 23:58:32.297521 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.297488    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"]
Jul 31 23:58:32.307277 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:58:32.307225    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 23:58:32.307277 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 23:58:32.307277 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 23:58:32.307277 ip-10-0-140-3 kubenswrapper[2578]:  > containerID="f443fb9c48779c2ef46e5d18a2efe4f8bc3e5c6fa5f062eaac0d426ec9dc5e32" cmd=["/bin/sleep","15"]
Jul 31 23:58:32.307710 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.307680    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq" message="" err=<
Jul 31 23:58:32.307710 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 23:58:32.307710 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 23:58:32.307710 ip-10-0-140-3 kubenswrapper[2578]:  >
Jul 31 23:58:32.308015 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:58:32.307732    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 23:58:32.308015 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 23:58:32.308015 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 23:58:32.308015 ip-10-0-140-3 kubenswrapper[2578]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq" podUID="b6212069-4342-485f-9c35-edfe74d025ea" containerName="main" containerID="cri-o://f443fb9c48779c2ef46e5d18a2efe4f8bc3e5c6fa5f062eaac0d426ec9dc5e32"
Jul 31 23:58:32.308015 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.307804    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq" podUID="b6212069-4342-485f-9c35-edfe74d025ea" containerName="main" containerID="cri-o://f443fb9c48779c2ef46e5d18a2efe4f8bc3e5c6fa5f062eaac0d426ec9dc5e32" gracePeriod=60
Jul 31 23:58:32.676260 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.676233    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:58:32.716992 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.716957    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-tmp-dir\") pod \"b6212069-4342-485f-9c35-edfe74d025ea\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") "
Jul 31 23:58:32.717196 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.717009    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-model-cache\") pod \"b6212069-4342-485f-9c35-edfe74d025ea\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") "
Jul 31 23:58:32.717196 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.717035    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6212069-4342-485f-9c35-edfe74d025ea-tls-certs\") pod \"b6212069-4342-485f-9c35-edfe74d025ea\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") "
Jul 31 23:58:32.717196 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.717077    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8799m\" (UniqueName: \"kubernetes.io/projected/b6212069-4342-485f-9c35-edfe74d025ea-kube-api-access-8799m\") pod \"b6212069-4342-485f-9c35-edfe74d025ea\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") "
Jul 31 23:58:32.717196 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.717115    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-dshm\") pod \"b6212069-4342-485f-9c35-edfe74d025ea\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") "
Jul 31 23:58:32.717196 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.717143    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-home\") pod \"b6212069-4342-485f-9c35-edfe74d025ea\" (UID: \"b6212069-4342-485f-9c35-edfe74d025ea\") "
Jul 31 23:58:32.717490 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.717209    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6212069-4342-485f-9c35-edfe74d025ea" (UID: "b6212069-4342-485f-9c35-edfe74d025ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:58:32.717490 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.717235    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6212069-4342-485f-9c35-edfe74d025ea" (UID: "b6212069-4342-485f-9c35-edfe74d025ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:58:32.717490 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.717426    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-tmp-dir\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:58:32.717490 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.717444    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-model-cache\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:58:32.717704 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.717658    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-home" (OuterVolumeSpecName: "home") pod "b6212069-4342-485f-9c35-edfe74d025ea" (UID: "b6212069-4342-485f-9c35-edfe74d025ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:58:32.720113 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.720087    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-dshm" (OuterVolumeSpecName: "dshm") pod "b6212069-4342-485f-9c35-edfe74d025ea" (UID: "b6212069-4342-485f-9c35-edfe74d025ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:58:32.720113 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.720093    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6212069-4342-485f-9c35-edfe74d025ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6212069-4342-485f-9c35-edfe74d025ea" (UID: "b6212069-4342-485f-9c35-edfe74d025ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:58:32.720310 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.720207    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6212069-4342-485f-9c35-edfe74d025ea-kube-api-access-8799m" (OuterVolumeSpecName: "kube-api-access-8799m") pod "b6212069-4342-485f-9c35-edfe74d025ea" (UID: "b6212069-4342-485f-9c35-edfe74d025ea"). InnerVolumeSpecName "kube-api-access-8799m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:58:32.818663 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.818630    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6212069-4342-485f-9c35-edfe74d025ea-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:58:32.818663 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.818661    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8799m\" (UniqueName: \"kubernetes.io/projected/b6212069-4342-485f-9c35-edfe74d025ea-kube-api-access-8799m\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:58:32.818663 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.818671    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-dshm\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:58:32.818899 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:32.818680    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6212069-4342-485f-9c35-edfe74d025ea-home\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:58:33.048013 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:33.047971    2578 generic.go:356] "Generic (PLEG): container finished" podID="b6212069-4342-485f-9c35-edfe74d025ea" containerID="f443fb9c48779c2ef46e5d18a2efe4f8bc3e5c6fa5f062eaac0d426ec9dc5e32" exitCode=0
Jul 31 23:58:33.048185 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:33.048023    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"
Jul 31 23:58:33.048185 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:33.048055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq" event={"ID":"b6212069-4342-485f-9c35-edfe74d025ea","Type":"ContainerDied","Data":"f443fb9c48779c2ef46e5d18a2efe4f8bc3e5c6fa5f062eaac0d426ec9dc5e32"}
Jul 31 23:58:33.048185 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:33.048108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq" event={"ID":"b6212069-4342-485f-9c35-edfe74d025ea","Type":"ContainerDied","Data":"20442cef99de494f7f0007b168123dc8b9e4e7c0500c6cf38f9f6b6969736e61"}
Jul 31 23:58:33.048185 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:33.048123    2578 scope.go:117] "RemoveContainer" containerID="f443fb9c48779c2ef46e5d18a2efe4f8bc3e5c6fa5f062eaac0d426ec9dc5e32"
Jul 31 23:58:33.057569 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:33.057548    2578 scope.go:117] "RemoveContainer" containerID="f443fb9c48779c2ef46e5d18a2efe4f8bc3e5c6fa5f062eaac0d426ec9dc5e32"
Jul 31 23:58:33.057856 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:58:33.057835    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f443fb9c48779c2ef46e5d18a2efe4f8bc3e5c6fa5f062eaac0d426ec9dc5e32\": container with ID starting with f443fb9c48779c2ef46e5d18a2efe4f8bc3e5c6fa5f062eaac0d426ec9dc5e32 not found: ID does not exist" containerID="f443fb9c48779c2ef46e5d18a2efe4f8bc3e5c6fa5f062eaac0d426ec9dc5e32"
Jul 31 23:58:33.057948 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:33.057863    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f443fb9c48779c2ef46e5d18a2efe4f8bc3e5c6fa5f062eaac0d426ec9dc5e32"} err="failed to get container status \"f443fb9c48779c2ef46e5d18a2efe4f8bc3e5c6fa5f062eaac0d426ec9dc5e32\": rpc error: code = NotFound desc = could not find container \"f443fb9c48779c2ef46e5d18a2efe4f8bc3e5c6fa5f062eaac0d426ec9dc5e32\": container with ID starting with f443fb9c48779c2ef46e5d18a2efe4f8bc3e5c6fa5f062eaac0d426ec9dc5e32 not found: ID does not exist"
Jul 31 23:58:33.086207 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:33.086179    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"]
Jul 31 23:58:33.097986 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:33.097963    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zxhgq"]
Jul 31 23:58:33.186149 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:58:33.186108    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6212069-4342-485f-9c35-edfe74d025ea" path="/var/lib/kubelet/pods/b6212069-4342-485f-9c35-edfe74d025ea/volumes"
Jul 31 23:59:10.207250 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.207214    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"]
Jul 31 23:59:10.207770 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.207632    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6212069-4342-485f-9c35-edfe74d025ea" containerName="main"
Jul 31 23:59:10.207770 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.207644    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6212069-4342-485f-9c35-edfe74d025ea" containerName="main"
Jul 31 23:59:10.207770 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.207707    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6212069-4342-485f-9c35-edfe74d025ea" containerName="main"
Jul 31 23:59:10.211926 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.211906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.214607 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.214584    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Jul 31 23:59:10.214607 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.214600    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-7j7k6\""
Jul 31 23:59:10.214816 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.214644    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Jul 31 23:59:10.214816 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.214694    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Jul 31 23:59:10.224230 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.224205    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"]
Jul 31 23:59:10.344025 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.343991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-home\") pod \"fc-concurrency-test-kserve-b5d44467c-lpjsf\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.344198 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.344052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-lpjsf\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.344198 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.344145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-lpjsf\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.344282 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.344228    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-lpjsf\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.344282 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.344263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r55cl\" (UniqueName: \"kubernetes.io/projected/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-kube-api-access-r55cl\") pod \"fc-concurrency-test-kserve-b5d44467c-lpjsf\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.344357 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.344304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-lpjsf\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.445441 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.445409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-lpjsf\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.445625 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.445459    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-lpjsf\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.445625 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.445479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r55cl\" (UniqueName: \"kubernetes.io/projected/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-kube-api-access-r55cl\") pod \"fc-concurrency-test-kserve-b5d44467c-lpjsf\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.445625 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.445509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-lpjsf\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.445625 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.445548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-home\") pod \"fc-concurrency-test-kserve-b5d44467c-lpjsf\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.445625 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.445594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-lpjsf\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.445942 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.445908    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-lpjsf\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.446039 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.445981    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-lpjsf\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.446185 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.446157    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-home\") pod \"fc-concurrency-test-kserve-b5d44467c-lpjsf\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.448258 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.447912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-lpjsf\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.448407 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.448385    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-lpjsf\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.459107 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.459036    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r55cl\" (UniqueName: \"kubernetes.io/projected/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-kube-api-access-r55cl\") pod \"fc-concurrency-test-kserve-b5d44467c-lpjsf\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.522892 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.522858    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:10.671663 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:10.671635    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"]
Jul 31 23:59:10.673842 ip-10-0-140-3 kubenswrapper[2578]: W0731 23:59:10.673816    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce18e2e_ac43_4200_8cbb_76ab4da95ec8.slice/crio-0c591494367427820a8024a00ed627a9db325f64918511e8fe2b6be8f19e58d5 WatchSource:0}: Error finding container 0c591494367427820a8024a00ed627a9db325f64918511e8fe2b6be8f19e58d5: Status 404 returned error can't find the container with id 0c591494367427820a8024a00ed627a9db325f64918511e8fe2b6be8f19e58d5
Jul 31 23:59:11.250776 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:11.250741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf" event={"ID":"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8","Type":"ContainerStarted","Data":"859590b555b1e2c3eb218efd8e39f1f451a1bfc5a55884ffd9c565c086fd0281"}
Jul 31 23:59:11.251175 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:11.250788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf" event={"ID":"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8","Type":"ContainerStarted","Data":"0c591494367427820a8024a00ed627a9db325f64918511e8fe2b6be8f19e58d5"}
Jul 31 23:59:11.287330 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:11.287278    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf" podStartSLOduration=1.287262895 podStartE2EDuration="1.287262895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:59:11.286361264 +0000 UTC m=+790.646252306" watchObservedRunningTime="2026-07-31 23:59:11.287262895 +0000 UTC m=+790.647153937"
Jul 31 23:59:11.523809 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:11.523709    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:20.523981 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:20.523946    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:20.536978 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:20.536955    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:21.323262 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:21.323227    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:30.145490 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:30.145453    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6"]
Jul 31 23:59:39.642503 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:39.642450    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6" podUID="8e6862a8-2da5-494c-9e5e-0a0da4ce9acf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 23:59:45.166440 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:45.166399    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6" podUID="8e6862a8-2da5-494c-9e5e-0a0da4ce9acf" containerName="main" containerID="cri-o://ee52536672307fe12174790705c64f188666e1fbe992077616f4655a0a0ac18d" gracePeriod=45
Jul 31 23:59:45.417523 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:45.417465    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6"
Jul 31 23:59:45.450656 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:45.450621    2578 generic.go:356] "Generic (PLEG): container finished" podID="8e6862a8-2da5-494c-9e5e-0a0da4ce9acf" containerID="ee52536672307fe12174790705c64f188666e1fbe992077616f4655a0a0ac18d" exitCode=0
Jul 31 23:59:45.450861 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:45.450695    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6"
Jul 31 23:59:45.450861 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:45.450702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6" event={"ID":"8e6862a8-2da5-494c-9e5e-0a0da4ce9acf","Type":"ContainerDied","Data":"ee52536672307fe12174790705c64f188666e1fbe992077616f4655a0a0ac18d"}
Jul 31 23:59:45.450861 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:45.450810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6" event={"ID":"8e6862a8-2da5-494c-9e5e-0a0da4ce9acf","Type":"ContainerDied","Data":"7b7020144283a02fe817803362290f544be66425d78597a286dea5a648e1d778"}
Jul 31 23:59:45.450861 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:45.450831    2578 scope.go:117] "RemoveContainer" containerID="ee52536672307fe12174790705c64f188666e1fbe992077616f4655a0a0ac18d"
Jul 31 23:59:45.460525 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:45.460505    2578 scope.go:117] "RemoveContainer" containerID="ee52536672307fe12174790705c64f188666e1fbe992077616f4655a0a0ac18d"
Jul 31 23:59:45.460831 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:59:45.460803    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee52536672307fe12174790705c64f188666e1fbe992077616f4655a0a0ac18d\": container with ID starting with ee52536672307fe12174790705c64f188666e1fbe992077616f4655a0a0ac18d not found: ID does not exist" containerID="ee52536672307fe12174790705c64f188666e1fbe992077616f4655a0a0ac18d"
Jul 31 23:59:45.460948 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:45.460840    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee52536672307fe12174790705c64f188666e1fbe992077616f4655a0a0ac18d"} err="failed to get container status \"ee52536672307fe12174790705c64f188666e1fbe992077616f4655a0a0ac18d\": rpc error: code = NotFound desc = could not find container \"ee52536672307fe12174790705c64f188666e1fbe992077616f4655a0a0ac18d\": container with ID starting with ee52536672307fe12174790705c64f188666e1fbe992077616f4655a0a0ac18d not found: ID does not exist"
Jul 31 23:59:45.550925 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:45.550893    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6862a8-2da5-494c-9e5e-0a0da4ce9acf-tls-certs\") pod \"8e6862a8-2da5-494c-9e5e-0a0da4ce9acf\" (UID: \"8e6862a8-2da5-494c-9e5e-0a0da4ce9acf\") "
Jul 31 23:59:45.551086 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:45.550940    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ndn9\" (UniqueName: \"kubernetes.io/projected/8e6862a8-2da5-494c-9e5e-0a0da4ce9acf-kube-api-access-4ndn9\") pod \"8e6862a8-2da5-494c-9e5e-0a0da4ce9acf\" (UID: \"8e6862a8-2da5-494c-9e5e-0a0da4ce9acf\") "
Jul 31 23:59:45.553282 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:45.553250    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6862a8-2da5-494c-9e5e-0a0da4ce9acf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e6862a8-2da5-494c-9e5e-0a0da4ce9acf" (UID: "8e6862a8-2da5-494c-9e5e-0a0da4ce9acf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:59:45.553282 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:45.553272    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6862a8-2da5-494c-9e5e-0a0da4ce9acf-kube-api-access-4ndn9" (OuterVolumeSpecName: "kube-api-access-4ndn9") pod "8e6862a8-2da5-494c-9e5e-0a0da4ce9acf" (UID: "8e6862a8-2da5-494c-9e5e-0a0da4ce9acf"). InnerVolumeSpecName "kube-api-access-4ndn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:59:45.651685 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:45.651650    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6862a8-2da5-494c-9e5e-0a0da4ce9acf-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:59:45.651685 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:45.651678    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ndn9\" (UniqueName: \"kubernetes.io/projected/8e6862a8-2da5-494c-9e5e-0a0da4ce9acf-kube-api-access-4ndn9\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:59:45.771476 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:45.771450    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6"]
Jul 31 23:59:45.773538 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:45.773518    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheb7cg6"]
Jul 31 23:59:47.185217 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:47.185183    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e6862a8-2da5-494c-9e5e-0a0da4ce9acf" path="/var/lib/kubelet/pods/8e6862a8-2da5-494c-9e5e-0a0da4ce9acf/volumes"
Jul 31 23:59:48.527390 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:48.527352    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"]
Jul 31 23:59:48.537300 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:59:48.537253    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 23:59:48.537300 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 23:59:48.537300 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 23:59:48.537300 ip-10-0-140-3 kubenswrapper[2578]:  > containerID="859590b555b1e2c3eb218efd8e39f1f451a1bfc5a55884ffd9c565c086fd0281" cmd=["/bin/sleep","15"]
Jul 31 23:59:48.537606 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:48.537325    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf" message="" err=<
Jul 31 23:59:48.537606 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 23:59:48.537606 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 23:59:48.537606 ip-10-0-140-3 kubenswrapper[2578]:  >
Jul 31 23:59:48.537606 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:59:48.537342    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 23:59:48.537606 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 23:59:48.537606 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 23:59:48.537606 ip-10-0-140-3 kubenswrapper[2578]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf" podUID="7ce18e2e-ac43-4200-8cbb-76ab4da95ec8" containerName="main" containerID="cri-o://859590b555b1e2c3eb218efd8e39f1f451a1bfc5a55884ffd9c565c086fd0281"
Jul 31 23:59:48.537606 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:48.537398    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf" podUID="7ce18e2e-ac43-4200-8cbb-76ab4da95ec8" containerName="main" containerID="cri-o://859590b555b1e2c3eb218efd8e39f1f451a1bfc5a55884ffd9c565c086fd0281" gracePeriod=60
Jul 31 23:59:48.896763 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:48.896738    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:48.981739 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:48.981690    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-tls-certs\") pod \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") "
Jul 31 23:59:48.981739 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:48.981746    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-dshm\") pod \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") "
Jul 31 23:59:48.981990 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:48.981783    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-tmp-dir\") pod \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") "
Jul 31 23:59:48.981990 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:48.981814    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-model-cache\") pod \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") "
Jul 31 23:59:48.981990 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:48.981859    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-home\") pod \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") "
Jul 31 23:59:48.981990 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:48.981907    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r55cl\" (UniqueName: \"kubernetes.io/projected/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-kube-api-access-r55cl\") pod \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\" (UID: \"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8\") "
Jul 31 23:59:48.982242 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:48.982081    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ce18e2e-ac43-4200-8cbb-76ab4da95ec8" (UID: "7ce18e2e-ac43-4200-8cbb-76ab4da95ec8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:59:48.982242 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:48.982087    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ce18e2e-ac43-4200-8cbb-76ab4da95ec8" (UID: "7ce18e2e-ac43-4200-8cbb-76ab4da95ec8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:59:48.982242 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:48.982156    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-home" (OuterVolumeSpecName: "home") pod "7ce18e2e-ac43-4200-8cbb-76ab4da95ec8" (UID: "7ce18e2e-ac43-4200-8cbb-76ab4da95ec8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:59:48.984142 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:48.984119    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-dshm" (OuterVolumeSpecName: "dshm") pod "7ce18e2e-ac43-4200-8cbb-76ab4da95ec8" (UID: "7ce18e2e-ac43-4200-8cbb-76ab4da95ec8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:59:48.984261 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:48.984186    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ce18e2e-ac43-4200-8cbb-76ab4da95ec8" (UID: "7ce18e2e-ac43-4200-8cbb-76ab4da95ec8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:59:48.984261 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:48.984235    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-kube-api-access-r55cl" (OuterVolumeSpecName: "kube-api-access-r55cl") pod "7ce18e2e-ac43-4200-8cbb-76ab4da95ec8" (UID: "7ce18e2e-ac43-4200-8cbb-76ab4da95ec8"). InnerVolumeSpecName "kube-api-access-r55cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:59:49.083476 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:49.083381    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-home\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:59:49.083476 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:49.083412    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r55cl\" (UniqueName: \"kubernetes.io/projected/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-kube-api-access-r55cl\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:59:49.083476 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:49.083427    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:59:49.083476 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:49.083439    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-dshm\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:59:49.083476 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:49.083450    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-tmp-dir\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:59:49.083476 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:49.083460    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8-model-cache\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Jul 31 23:59:49.484212 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:49.484175    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ce18e2e-ac43-4200-8cbb-76ab4da95ec8" containerID="859590b555b1e2c3eb218efd8e39f1f451a1bfc5a55884ffd9c565c086fd0281" exitCode=0
Jul 31 23:59:49.484400 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:49.484246    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"
Jul 31 23:59:49.484400 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:49.484258    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf" event={"ID":"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8","Type":"ContainerDied","Data":"859590b555b1e2c3eb218efd8e39f1f451a1bfc5a55884ffd9c565c086fd0281"}
Jul 31 23:59:49.484400 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:49.484336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf" event={"ID":"7ce18e2e-ac43-4200-8cbb-76ab4da95ec8","Type":"ContainerDied","Data":"0c591494367427820a8024a00ed627a9db325f64918511e8fe2b6be8f19e58d5"}
Jul 31 23:59:49.484400 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:49.484358    2578 scope.go:117] "RemoveContainer" containerID="859590b555b1e2c3eb218efd8e39f1f451a1bfc5a55884ffd9c565c086fd0281"
Jul 31 23:59:49.493960 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:49.493939    2578 scope.go:117] "RemoveContainer" containerID="859590b555b1e2c3eb218efd8e39f1f451a1bfc5a55884ffd9c565c086fd0281"
Jul 31 23:59:49.494201 ip-10-0-140-3 kubenswrapper[2578]: E0731 23:59:49.494183    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"859590b555b1e2c3eb218efd8e39f1f451a1bfc5a55884ffd9c565c086fd0281\": container with ID starting with 859590b555b1e2c3eb218efd8e39f1f451a1bfc5a55884ffd9c565c086fd0281 not found: ID does not exist" containerID="859590b555b1e2c3eb218efd8e39f1f451a1bfc5a55884ffd9c565c086fd0281"
Jul 31 23:59:49.494266 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:49.494208    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859590b555b1e2c3eb218efd8e39f1f451a1bfc5a55884ffd9c565c086fd0281"} err="failed to get container status \"859590b555b1e2c3eb218efd8e39f1f451a1bfc5a55884ffd9c565c086fd0281\": rpc error: code = NotFound desc = could not find container \"859590b555b1e2c3eb218efd8e39f1f451a1bfc5a55884ffd9c565c086fd0281\": container with ID starting with 859590b555b1e2c3eb218efd8e39f1f451a1bfc5a55884ffd9c565c086fd0281 not found: ID does not exist"
Jul 31 23:59:49.500859 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:49.500833    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"]
Jul 31 23:59:49.505871 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:49.505850    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lpjsf"]
Jul 31 23:59:51.185340 ip-10-0-140-3 kubenswrapper[2578]: I0731 23:59:51.185301    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ce18e2e-ac43-4200-8cbb-76ab4da95ec8" path="/var/lib/kubelet/pods/7ce18e2e-ac43-4200-8cbb-76ab4da95ec8/volumes"
Aug 01 00:00:27.643398 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.643357    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"]
Aug 01 00:00:27.643984 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.643964    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ce18e2e-ac43-4200-8cbb-76ab4da95ec8" containerName="main"
Aug 01 00:00:27.644066 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.643989    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ce18e2e-ac43-4200-8cbb-76ab4da95ec8" containerName="main"
Aug 01 00:00:27.644066 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.644013    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e6862a8-2da5-494c-9e5e-0a0da4ce9acf" containerName="main"
Aug 01 00:00:27.644066 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.644021    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e6862a8-2da5-494c-9e5e-0a0da4ce9acf" containerName="main"
Aug 01 00:00:27.644230 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.644122    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ce18e2e-ac43-4200-8cbb-76ab4da95ec8" containerName="main"
Aug 01 00:00:27.644230 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.644139    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e6862a8-2da5-494c-9e5e-0a0da4ce9acf" containerName="main"
Aug 01 00:00:27.647306 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.647284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.650463 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.650443    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-c8qrn\""
Aug 01 00:00:27.651129 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.651107    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Aug 01 00:00:27.651231 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.651114    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Aug 01 00:00:27.651231 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.651169    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Aug 01 00:00:27.672319 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.672285    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"]
Aug 01 00:00:27.712573 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.712535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.712795 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.712594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.712795 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.712620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.712795 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.712653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.712795 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.712679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmnvq\" (UniqueName: \"kubernetes.io/projected/faea06ba-1f15-46d8-82bb-4cf72696e3df-kube-api-access-bmnvq\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.712795 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.712705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.712795 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.712749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faea06ba-1f15-46d8-82bb-4cf72696e3df-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.712795 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.712776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.713151 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.712837    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.813896 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.813857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.814073 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.813961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.814073 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.813993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.814073 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.814009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.814202 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.814148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.814202 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.814194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmnvq\" (UniqueName: \"kubernetes.io/projected/faea06ba-1f15-46d8-82bb-4cf72696e3df-kube-api-access-bmnvq\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.814370 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.814337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.814494 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.814399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faea06ba-1f15-46d8-82bb-4cf72696e3df-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.814494 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.814404    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.814494 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.814426    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.814494 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.814430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.814740 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.814524    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.814815 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.814740    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.815077 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.815055    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faea06ba-1f15-46d8-82bb-4cf72696e3df-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.816402 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.816377    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.816614 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.816594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.824347 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.824323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.824563 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.824541    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmnvq\" (UniqueName: \"kubernetes.io/projected/faea06ba-1f15-46d8-82bb-4cf72696e3df-kube-api-access-bmnvq\") pod \"router-gateway-1-openshift-default-66bbccc5fb-2vcrb\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:27.959016 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:27.958976    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:28.100965 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:28.100942    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"]
Aug 01 00:00:28.102879 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:00:28.102849    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaea06ba_1f15_46d8_82bb_4cf72696e3df.slice/crio-2ea640ca8a96546ef36ed0482d4eff89d568c248dd679bded187ef04a59056e1 WatchSource:0}: Error finding container 2ea640ca8a96546ef36ed0482d4eff89d568c248dd679bded187ef04a59056e1: Status 404 returned error can't find the container with id 2ea640ca8a96546ef36ed0482d4eff89d568c248dd679bded187ef04a59056e1
Aug 01 00:00:28.714722 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:28.714674    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb" event={"ID":"faea06ba-1f15-46d8-82bb-4cf72696e3df","Type":"ContainerStarted","Data":"2ea640ca8a96546ef36ed0482d4eff89d568c248dd679bded187ef04a59056e1"}
Aug 01 00:00:33.484581 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:33.484547    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Aug 01 00:00:33.484900 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:33.484611    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Aug 01 00:00:33.484900 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:33.484637    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Aug 01 00:00:33.750145 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:33.750044    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb" event={"ID":"faea06ba-1f15-46d8-82bb-4cf72696e3df","Type":"ContainerStarted","Data":"58173ce45b155e153b194beb62777d6af2323f7917ab0f26cfb689da09248a5e"}
Aug 01 00:00:33.770847 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:33.770800    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb" podStartSLOduration=1.391670697 podStartE2EDuration="6.770783897s" totalImagesPullingTime="5.3791132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:00:33.769633228 +0000 UTC m=+873.129524269" watchObservedRunningTime="2026-08-01 00:00:33.770783897 +0000 UTC m=+873.130674937"
Aug 01 00:00:33.959629 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:33.959594    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:34.965210 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:34.965175    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:35.761746 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:35.761703    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:00:35.762863 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:00:35.762828    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:01:01.163573 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:01.163542    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:01:01.164101 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:01.163616    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:01:01.187101 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:01.187066    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"]
Aug 01 00:01:01.187393 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:01.187343    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb" podUID="faea06ba-1f15-46d8-82bb-4cf72696e3df" containerName="istio-proxy" containerID="cri-o://58173ce45b155e153b194beb62777d6af2323f7917ab0f26cfb689da09248a5e" gracePeriod=30
Aug 01 00:01:02.909377 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:02.909332    2578 generic.go:356] "Generic (PLEG): container finished" podID="faea06ba-1f15-46d8-82bb-4cf72696e3df" containerID="58173ce45b155e153b194beb62777d6af2323f7917ab0f26cfb689da09248a5e" exitCode=0
Aug 01 00:01:02.909743 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:02.909399    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb" event={"ID":"faea06ba-1f15-46d8-82bb-4cf72696e3df","Type":"ContainerDied","Data":"58173ce45b155e153b194beb62777d6af2323f7917ab0f26cfb689da09248a5e"}
Aug 01 00:01:03.052874 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.052849    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:01:03.129172 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.128889    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmnvq\" (UniqueName: \"kubernetes.io/projected/faea06ba-1f15-46d8-82bb-4cf72696e3df-kube-api-access-bmnvq\") pod \"faea06ba-1f15-46d8-82bb-4cf72696e3df\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") "
Aug 01 00:01:03.129172 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.128968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-workload-socket\") pod \"faea06ba-1f15-46d8-82bb-4cf72696e3df\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") "
Aug 01 00:01:03.129172 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.129077    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-workload-certs\") pod \"faea06ba-1f15-46d8-82bb-4cf72696e3df\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") "
Aug 01 00:01:03.129172 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.129103    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-token\") pod \"faea06ba-1f15-46d8-82bb-4cf72696e3df\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") "
Aug 01 00:01:03.129172 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.129128    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-envoy\") pod \"faea06ba-1f15-46d8-82bb-4cf72696e3df\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") "
Aug 01 00:01:03.129172 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.129155    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faea06ba-1f15-46d8-82bb-4cf72696e3df-istiod-ca-cert\") pod \"faea06ba-1f15-46d8-82bb-4cf72696e3df\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") "
Aug 01 00:01:03.129597 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.129188    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-credential-socket\") pod \"faea06ba-1f15-46d8-82bb-4cf72696e3df\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") "
Aug 01 00:01:03.129597 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.129243    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-podinfo\") pod \"faea06ba-1f15-46d8-82bb-4cf72696e3df\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") "
Aug 01 00:01:03.129597 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.129281    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-data\") pod \"faea06ba-1f15-46d8-82bb-4cf72696e3df\" (UID: \"faea06ba-1f15-46d8-82bb-4cf72696e3df\") "
Aug 01 00:01:03.129597 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.129328    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "faea06ba-1f15-46d8-82bb-4cf72696e3df" (UID: "faea06ba-1f15-46d8-82bb-4cf72696e3df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:01:03.129597 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.129553    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-workload-socket\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:01:03.129918 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.129783    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "faea06ba-1f15-46d8-82bb-4cf72696e3df" (UID: "faea06ba-1f15-46d8-82bb-4cf72696e3df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:01:03.130263 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.130109    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faea06ba-1f15-46d8-82bb-4cf72696e3df-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "faea06ba-1f15-46d8-82bb-4cf72696e3df" (UID: "faea06ba-1f15-46d8-82bb-4cf72696e3df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:01:03.130263 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.130149    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "faea06ba-1f15-46d8-82bb-4cf72696e3df" (UID: "faea06ba-1f15-46d8-82bb-4cf72696e3df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:01:03.130449 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.130308    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-data" (OuterVolumeSpecName: "istio-data") pod "faea06ba-1f15-46d8-82bb-4cf72696e3df" (UID: "faea06ba-1f15-46d8-82bb-4cf72696e3df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:01:03.132937 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.132817    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "faea06ba-1f15-46d8-82bb-4cf72696e3df" (UID: "faea06ba-1f15-46d8-82bb-4cf72696e3df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:01:03.132937 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.132911    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-token" (OuterVolumeSpecName: "istio-token") pod "faea06ba-1f15-46d8-82bb-4cf72696e3df" (UID: "faea06ba-1f15-46d8-82bb-4cf72696e3df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:01:03.133111 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.133022    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faea06ba-1f15-46d8-82bb-4cf72696e3df-kube-api-access-bmnvq" (OuterVolumeSpecName: "kube-api-access-bmnvq") pod "faea06ba-1f15-46d8-82bb-4cf72696e3df" (UID: "faea06ba-1f15-46d8-82bb-4cf72696e3df"). InnerVolumeSpecName "kube-api-access-bmnvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:01:03.133257 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.133231    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "faea06ba-1f15-46d8-82bb-4cf72696e3df" (UID: "faea06ba-1f15-46d8-82bb-4cf72696e3df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 01 00:01:03.231004 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.230970    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-envoy\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:01:03.231004 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.230997    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faea06ba-1f15-46d8-82bb-4cf72696e3df-istiod-ca-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:01:03.231004 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.231006    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-credential-socket\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:01:03.231004 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.231014    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-podinfo\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:01:03.231273 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.231023    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-data\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:01:03.231273 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.231035    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmnvq\" (UniqueName: \"kubernetes.io/projected/faea06ba-1f15-46d8-82bb-4cf72696e3df-kube-api-access-bmnvq\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:01:03.231273 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.231049    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faea06ba-1f15-46d8-82bb-4cf72696e3df-workload-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:01:03.231273 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.231057    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faea06ba-1f15-46d8-82bb-4cf72696e3df-istio-token\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:01:03.917125 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.917089    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb" event={"ID":"faea06ba-1f15-46d8-82bb-4cf72696e3df","Type":"ContainerDied","Data":"2ea640ca8a96546ef36ed0482d4eff89d568c248dd679bded187ef04a59056e1"}
Aug 01 00:01:03.917584 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.917155    2578 scope.go:117] "RemoveContainer" containerID="58173ce45b155e153b194beb62777d6af2323f7917ab0f26cfb689da09248a5e"
Aug 01 00:01:03.917584 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.917100    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"
Aug 01 00:01:03.935707 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.935681    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"]
Aug 01 00:01:03.937900 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:03.937873    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-2vcrb"]
Aug 01 00:01:05.185793 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:05.185759    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faea06ba-1f15-46d8-82bb-4cf72696e3df" path="/var/lib/kubelet/pods/faea06ba-1f15-46d8-82bb-4cf72696e3df/volumes"
Aug 01 00:01:18.467483 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.467452    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8"]
Aug 01 00:01:18.467888 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.467863    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faea06ba-1f15-46d8-82bb-4cf72696e3df" containerName="istio-proxy"
Aug 01 00:01:18.467888 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.467875    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="faea06ba-1f15-46d8-82bb-4cf72696e3df" containerName="istio-proxy"
Aug 01 00:01:18.467965 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.467945    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="faea06ba-1f15-46d8-82bb-4cf72696e3df" containerName="istio-proxy"
Aug 01 00:01:18.476225 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.476192    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8"
Aug 01 00:01:18.478239 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.478217    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Aug 01 00:01:18.478798 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.478691    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Aug 01 00:01:18.478798 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.478703    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-jsbbp\""
Aug 01 00:01:18.478798 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.478781    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Aug 01 00:01:18.479085 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.478848    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-vx5qj\""
Aug 01 00:01:18.480005 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.479980    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8"]
Aug 01 00:01:18.569745 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.569683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbt5l\" (UniqueName: \"kubernetes.io/projected/4d877c2c-1cb5-4370-8d03-7e72706e3082-kube-api-access-kbt5l\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8\" (UID: \"4d877c2c-1cb5-4370-8d03-7e72706e3082\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8"
Aug 01 00:01:18.569910 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.569785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d877c2c-1cb5-4370-8d03-7e72706e3082-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8\" (UID: \"4d877c2c-1cb5-4370-8d03-7e72706e3082\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8"
Aug 01 00:01:18.671153 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.671115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbt5l\" (UniqueName: \"kubernetes.io/projected/4d877c2c-1cb5-4370-8d03-7e72706e3082-kube-api-access-kbt5l\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8\" (UID: \"4d877c2c-1cb5-4370-8d03-7e72706e3082\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8"
Aug 01 00:01:18.671359 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.671173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d877c2c-1cb5-4370-8d03-7e72706e3082-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8\" (UID: \"4d877c2c-1cb5-4370-8d03-7e72706e3082\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8"
Aug 01 00:01:18.673896 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.673872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d877c2c-1cb5-4370-8d03-7e72706e3082-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8\" (UID: \"4d877c2c-1cb5-4370-8d03-7e72706e3082\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8"
Aug 01 00:01:18.680392 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.680367    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbt5l\" (UniqueName: \"kubernetes.io/projected/4d877c2c-1cb5-4370-8d03-7e72706e3082-kube-api-access-kbt5l\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8\" (UID: \"4d877c2c-1cb5-4370-8d03-7e72706e3082\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8"
Aug 01 00:01:18.789410 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.789315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8"
Aug 01 00:01:18.926994 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:18.926967    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8"]
Aug 01 00:01:18.930028 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:01:18.930001    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d877c2c_1cb5_4370_8d03_7e72706e3082.slice/crio-a3ff60f40f830e0caa483dcee239f251529df30236012d5798ec786adbf9878c WatchSource:0}: Error finding container a3ff60f40f830e0caa483dcee239f251529df30236012d5798ec786adbf9878c: Status 404 returned error can't find the container with id a3ff60f40f830e0caa483dcee239f251529df30236012d5798ec786adbf9878c
Aug 01 00:01:19.007011 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:19.006965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8" event={"ID":"4d877c2c-1cb5-4370-8d03-7e72706e3082","Type":"ContainerStarted","Data":"e2cb8c61466287075e7e5b1c3335ebbea9398a04c8468c29087c0bfab843b168"}
Aug 01 00:01:19.007184 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:19.007025    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8" event={"ID":"4d877c2c-1cb5-4370-8d03-7e72706e3082","Type":"ContainerStarted","Data":"a3ff60f40f830e0caa483dcee239f251529df30236012d5798ec786adbf9878c"}
Aug 01 00:01:19.007184 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:19.007088    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8"
Aug 01 00:01:19.023214 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:19.023153    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8" podStartSLOduration=1.02313281 podStartE2EDuration="1.02313281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:01:19.019909518 +0000 UTC m=+918.379800579" watchObservedRunningTime="2026-08-01 00:01:19.02313281 +0000 UTC m=+918.383023851"
Aug 01 00:01:50.014979 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:01:50.014947    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8"
Aug 01 00:03:25.400842 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:25.400795    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8"]
Aug 01 00:03:30.014356 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:30.014318    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8" podUID="4d877c2c-1cb5-4370-8d03-7e72706e3082" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:03:40.014915 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:40.014877    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8" podUID="4d877c2c-1cb5-4370-8d03-7e72706e3082" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:03:40.423500 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:40.423446    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8" podUID="4d877c2c-1cb5-4370-8d03-7e72706e3082" containerName="main" containerID="cri-o://e2cb8c61466287075e7e5b1c3335ebbea9398a04c8468c29087c0bfab843b168" gracePeriod=45
Aug 01 00:03:40.682865 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:40.682808    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8"
Aug 01 00:03:40.780734 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:40.780685    2578 generic.go:356] "Generic (PLEG): container finished" podID="4d877c2c-1cb5-4370-8d03-7e72706e3082" containerID="e2cb8c61466287075e7e5b1c3335ebbea9398a04c8468c29087c0bfab843b168" exitCode=0
Aug 01 00:03:40.780923 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:40.780760    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8"
Aug 01 00:03:40.780923 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:40.780774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8" event={"ID":"4d877c2c-1cb5-4370-8d03-7e72706e3082","Type":"ContainerDied","Data":"e2cb8c61466287075e7e5b1c3335ebbea9398a04c8468c29087c0bfab843b168"}
Aug 01 00:03:40.780923 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:40.780834    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8" event={"ID":"4d877c2c-1cb5-4370-8d03-7e72706e3082","Type":"ContainerDied","Data":"a3ff60f40f830e0caa483dcee239f251529df30236012d5798ec786adbf9878c"}
Aug 01 00:03:40.780923 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:40.780862    2578 scope.go:117] "RemoveContainer" containerID="e2cb8c61466287075e7e5b1c3335ebbea9398a04c8468c29087c0bfab843b168"
Aug 01 00:03:40.789695 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:40.789678    2578 scope.go:117] "RemoveContainer" containerID="e2cb8c61466287075e7e5b1c3335ebbea9398a04c8468c29087c0bfab843b168"
Aug 01 00:03:40.789965 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:03:40.789948    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2cb8c61466287075e7e5b1c3335ebbea9398a04c8468c29087c0bfab843b168\": container with ID starting with e2cb8c61466287075e7e5b1c3335ebbea9398a04c8468c29087c0bfab843b168 not found: ID does not exist" containerID="e2cb8c61466287075e7e5b1c3335ebbea9398a04c8468c29087c0bfab843b168"
Aug 01 00:03:40.790037 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:40.789977    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2cb8c61466287075e7e5b1c3335ebbea9398a04c8468c29087c0bfab843b168"} err="failed to get container status \"e2cb8c61466287075e7e5b1c3335ebbea9398a04c8468c29087c0bfab843b168\": rpc error: code = NotFound desc = could not find container \"e2cb8c61466287075e7e5b1c3335ebbea9398a04c8468c29087c0bfab843b168\": container with ID starting with e2cb8c61466287075e7e5b1c3335ebbea9398a04c8468c29087c0bfab843b168 not found: ID does not exist"
Aug 01 00:03:40.805329 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:40.805303    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d877c2c-1cb5-4370-8d03-7e72706e3082-tls-certs\") pod \"4d877c2c-1cb5-4370-8d03-7e72706e3082\" (UID: \"4d877c2c-1cb5-4370-8d03-7e72706e3082\") "
Aug 01 00:03:40.805439 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:40.805363    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbt5l\" (UniqueName: \"kubernetes.io/projected/4d877c2c-1cb5-4370-8d03-7e72706e3082-kube-api-access-kbt5l\") pod \"4d877c2c-1cb5-4370-8d03-7e72706e3082\" (UID: \"4d877c2c-1cb5-4370-8d03-7e72706e3082\") "
Aug 01 00:03:40.807534 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:40.807512    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d877c2c-1cb5-4370-8d03-7e72706e3082-kube-api-access-kbt5l" (OuterVolumeSpecName: "kube-api-access-kbt5l") pod "4d877c2c-1cb5-4370-8d03-7e72706e3082" (UID: "4d877c2c-1cb5-4370-8d03-7e72706e3082"). InnerVolumeSpecName "kube-api-access-kbt5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:03:40.807638 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:40.807540    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d877c2c-1cb5-4370-8d03-7e72706e3082-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d877c2c-1cb5-4370-8d03-7e72706e3082" (UID: "4d877c2c-1cb5-4370-8d03-7e72706e3082"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:03:40.906666 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:40.906637    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbt5l\" (UniqueName: \"kubernetes.io/projected/4d877c2c-1cb5-4370-8d03-7e72706e3082-kube-api-access-kbt5l\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:03:40.906666 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:40.906664    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d877c2c-1cb5-4370-8d03-7e72706e3082-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:03:41.100883 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:41.100856    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8"]
Aug 01 00:03:41.104611 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:41.104581    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-d9dz8"]
Aug 01 00:03:41.185676 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:41.185647    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d877c2c-1cb5-4370-8d03-7e72706e3082" path="/var/lib/kubelet/pods/4d877c2c-1cb5-4370-8d03-7e72706e3082/volumes"
Aug 01 00:03:43.974756 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:43.974707    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr"]
Aug 01 00:03:43.975292 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:43.975271    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d877c2c-1cb5-4370-8d03-7e72706e3082" containerName="main"
Aug 01 00:03:43.975392 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:43.975294    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d877c2c-1cb5-4370-8d03-7e72706e3082" containerName="main"
Aug 01 00:03:43.975458 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:43.975406    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d877c2c-1cb5-4370-8d03-7e72706e3082" containerName="main"
Aug 01 00:03:43.979822 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:43.979805    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr"
Aug 01 00:03:43.981792 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:43.981760    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Aug 01 00:03:43.981912 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:43.981816    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Aug 01 00:03:43.982190 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:43.982174    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-nd89l\""
Aug 01 00:03:43.982271 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:43.982245    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-nm7mc\""
Aug 01 00:03:43.982330 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:43.982277    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Aug 01 00:03:43.987619 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:43.987595    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr"]
Aug 01 00:03:44.035351 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:44.035312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e43cedad-69c3-483c-814f-5e5d4e50bec6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr\" (UID: \"e43cedad-69c3-483c-814f-5e5d4e50bec6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr"
Aug 01 00:03:44.035530 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:44.035430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sscfv\" (UniqueName: \"kubernetes.io/projected/e43cedad-69c3-483c-814f-5e5d4e50bec6-kube-api-access-sscfv\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr\" (UID: \"e43cedad-69c3-483c-814f-5e5d4e50bec6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr"
Aug 01 00:03:44.136384 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:44.136349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sscfv\" (UniqueName: \"kubernetes.io/projected/e43cedad-69c3-483c-814f-5e5d4e50bec6-kube-api-access-sscfv\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr\" (UID: \"e43cedad-69c3-483c-814f-5e5d4e50bec6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr"
Aug 01 00:03:44.136541 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:44.136406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e43cedad-69c3-483c-814f-5e5d4e50bec6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr\" (UID: \"e43cedad-69c3-483c-814f-5e5d4e50bec6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr"
Aug 01 00:03:44.138923 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:44.138894    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e43cedad-69c3-483c-814f-5e5d4e50bec6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr\" (UID: \"e43cedad-69c3-483c-814f-5e5d4e50bec6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr"
Aug 01 00:03:44.143499 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:44.143470    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sscfv\" (UniqueName: \"kubernetes.io/projected/e43cedad-69c3-483c-814f-5e5d4e50bec6-kube-api-access-sscfv\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr\" (UID: \"e43cedad-69c3-483c-814f-5e5d4e50bec6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr"
Aug 01 00:03:44.293376 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:44.293278    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr"
Aug 01 00:03:44.434130 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:44.434102    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr"]
Aug 01 00:03:44.436496 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:03:44.436465    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43cedad_69c3_483c_814f_5e5d4e50bec6.slice/crio-06e14f6eaa5901979581570399633475e56d8b529a94bb3e28bf6ba2bbd7319d WatchSource:0}: Error finding container 06e14f6eaa5901979581570399633475e56d8b529a94bb3e28bf6ba2bbd7319d: Status 404 returned error can't find the container with id 06e14f6eaa5901979581570399633475e56d8b529a94bb3e28bf6ba2bbd7319d
Aug 01 00:03:44.806587 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:44.806543    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr" event={"ID":"e43cedad-69c3-483c-814f-5e5d4e50bec6","Type":"ContainerStarted","Data":"c8f12f4bf538e8240802bf7842c3e5ddbd1d116b52e8bd54fa748f69b4698715"}
Aug 01 00:03:44.806834 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:44.806605    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr" event={"ID":"e43cedad-69c3-483c-814f-5e5d4e50bec6","Type":"ContainerStarted","Data":"06e14f6eaa5901979581570399633475e56d8b529a94bb3e28bf6ba2bbd7319d"}
Aug 01 00:03:44.806834 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:44.806689    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr"
Aug 01 00:03:44.823159 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:03:44.823113    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr" podStartSLOduration=1.823097202 podStartE2EDuration="1.823097202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:03:44.82008956 +0000 UTC m=+1064.179980600" watchObservedRunningTime="2026-08-01 00:03:44.823097202 +0000 UTC m=+1064.182988242"
Aug 01 00:04:15.814218 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:15.814171    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr" podUID="e43cedad-69c3-483c-814f-5e5d4e50bec6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:04:25.813556 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:25.813520    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr" podUID="e43cedad-69c3-483c-814f-5e5d4e50bec6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:04:29.960603 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:29.960565    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr"]
Aug 01 00:04:29.962311 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:29.962280    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr" podUID="e43cedad-69c3-483c-814f-5e5d4e50bec6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:04:39.962638 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:39.962600    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr" podUID="e43cedad-69c3-483c-814f-5e5d4e50bec6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:04:44.980612 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:44.980544    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr" podUID="e43cedad-69c3-483c-814f-5e5d4e50bec6" containerName="main" containerID="cri-o://c8f12f4bf538e8240802bf7842c3e5ddbd1d116b52e8bd54fa748f69b4698715" gracePeriod=45
Aug 01 00:04:45.133649 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:45.133618    2578 generic.go:356] "Generic (PLEG): container finished" podID="e43cedad-69c3-483c-814f-5e5d4e50bec6" containerID="c8f12f4bf538e8240802bf7842c3e5ddbd1d116b52e8bd54fa748f69b4698715" exitCode=0
Aug 01 00:04:45.133827 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:45.133696    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr" event={"ID":"e43cedad-69c3-483c-814f-5e5d4e50bec6","Type":"ContainerDied","Data":"c8f12f4bf538e8240802bf7842c3e5ddbd1d116b52e8bd54fa748f69b4698715"}
Aug 01 00:04:45.241821 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:45.241749    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr"
Aug 01 00:04:45.360414 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:45.360378    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sscfv\" (UniqueName: \"kubernetes.io/projected/e43cedad-69c3-483c-814f-5e5d4e50bec6-kube-api-access-sscfv\") pod \"e43cedad-69c3-483c-814f-5e5d4e50bec6\" (UID: \"e43cedad-69c3-483c-814f-5e5d4e50bec6\") "
Aug 01 00:04:45.360607 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:45.360441    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e43cedad-69c3-483c-814f-5e5d4e50bec6-tls-certs\") pod \"e43cedad-69c3-483c-814f-5e5d4e50bec6\" (UID: \"e43cedad-69c3-483c-814f-5e5d4e50bec6\") "
Aug 01 00:04:45.362775 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:45.362748    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43cedad-69c3-483c-814f-5e5d4e50bec6-kube-api-access-sscfv" (OuterVolumeSpecName: "kube-api-access-sscfv") pod "e43cedad-69c3-483c-814f-5e5d4e50bec6" (UID: "e43cedad-69c3-483c-814f-5e5d4e50bec6"). InnerVolumeSpecName "kube-api-access-sscfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:04:45.362997 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:45.362969    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e43cedad-69c3-483c-814f-5e5d4e50bec6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e43cedad-69c3-483c-814f-5e5d4e50bec6" (UID: "e43cedad-69c3-483c-814f-5e5d4e50bec6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:04:45.460997 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:45.460963    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sscfv\" (UniqueName: \"kubernetes.io/projected/e43cedad-69c3-483c-814f-5e5d4e50bec6-kube-api-access-sscfv\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:04:45.460997 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:45.460995    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e43cedad-69c3-483c-814f-5e5d4e50bec6-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:04:46.140780 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:46.140751    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr"
Aug 01 00:04:46.141286 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:46.140747    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr" event={"ID":"e43cedad-69c3-483c-814f-5e5d4e50bec6","Type":"ContainerDied","Data":"06e14f6eaa5901979581570399633475e56d8b529a94bb3e28bf6ba2bbd7319d"}
Aug 01 00:04:46.141286 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:46.140882    2578 scope.go:117] "RemoveContainer" containerID="c8f12f4bf538e8240802bf7842c3e5ddbd1d116b52e8bd54fa748f69b4698715"
Aug 01 00:04:46.160695 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:46.160666    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr"]
Aug 01 00:04:46.164678 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:46.164657    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnvhqr"]
Aug 01 00:04:47.185459 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:04:47.185418    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e43cedad-69c3-483c-814f-5e5d4e50bec6" path="/var/lib/kubelet/pods/e43cedad-69c3-483c-814f-5e5d4e50bec6/volumes"
Aug 01 00:05:01.068855 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.068808    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"]
Aug 01 00:05:01.069354 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.069199    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e43cedad-69c3-483c-814f-5e5d4e50bec6" containerName="main"
Aug 01 00:05:01.069354 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.069211    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e43cedad-69c3-483c-814f-5e5d4e50bec6" containerName="main"
Aug 01 00:05:01.069354 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.069285    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e43cedad-69c3-483c-814f-5e5d4e50bec6" containerName="main"
Aug 01 00:05:01.072481 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.072461    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.074501 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.074481    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Aug 01 00:05:01.075234 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.075189    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Aug 01 00:05:01.075349 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.075212    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Aug 01 00:05:01.075349 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.075246    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-lcdzv\""
Aug 01 00:05:01.084162 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.084139    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"]
Aug 01 00:05:01.198850 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.198815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-w5gvk\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.198850 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.198850    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-w5gvk\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.199060 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.198910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6103ee63-337a-429f-a3bc-018d7244b50d-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-w5gvk\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.199060 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.198928    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-w5gvk\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.199060 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.198942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hj5m\" (UniqueName: \"kubernetes.io/projected/6103ee63-337a-429f-a3bc-018d7244b50d-kube-api-access-5hj5m\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-w5gvk\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.199060 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.199034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-w5gvk\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.299841 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.299803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-w5gvk\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.299841 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.299846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-w5gvk\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.300055 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.299898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6103ee63-337a-429f-a3bc-018d7244b50d-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-w5gvk\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.300055 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.299914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-w5gvk\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.300055 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.299929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hj5m\" (UniqueName: \"kubernetes.io/projected/6103ee63-337a-429f-a3bc-018d7244b50d-kube-api-access-5hj5m\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-w5gvk\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.300055 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.299994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-w5gvk\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.300275 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.300247    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-w5gvk\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.300275 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.300261    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-w5gvk\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.300391 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.300369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-w5gvk\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.302486 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.302461    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-w5gvk\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.302745 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.302708    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6103ee63-337a-429f-a3bc-018d7244b50d-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-w5gvk\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.306979 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.306960    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hj5m\" (UniqueName: \"kubernetes.io/projected/6103ee63-337a-429f-a3bc-018d7244b50d-kube-api-access-5hj5m\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-w5gvk\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.385653 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.385569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:01.527824 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:01.527673    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"]
Aug 01 00:05:01.531155 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:05:01.531126    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6103ee63_337a_429f_a3bc_018d7244b50d.slice/crio-eee87c3bfd9e328b15613b7de5fb4790c8518335caa574149b2442ad69fe7bbe WatchSource:0}: Error finding container eee87c3bfd9e328b15613b7de5fb4790c8518335caa574149b2442ad69fe7bbe: Status 404 returned error can't find the container with id eee87c3bfd9e328b15613b7de5fb4790c8518335caa574149b2442ad69fe7bbe
Aug 01 00:05:02.235235 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:02.235198    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk" event={"ID":"6103ee63-337a-429f-a3bc-018d7244b50d","Type":"ContainerStarted","Data":"93a00dbfd010d5e9b1cf6cea80a8e13eecc3aa01bb9d561933da7f7a69d97137"}
Aug 01 00:05:02.235628 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:02.235276    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk" event={"ID":"6103ee63-337a-429f-a3bc-018d7244b50d","Type":"ContainerStarted","Data":"eee87c3bfd9e328b15613b7de5fb4790c8518335caa574149b2442ad69fe7bbe"}
Aug 01 00:05:02.255066 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:02.255008    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk" podStartSLOduration=1.254990056 podStartE2EDuration="1.254990056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:05:02.253041239 +0000 UTC m=+1141.612932293" watchObservedRunningTime="2026-08-01 00:05:02.254990056 +0000 UTC m=+1141.614881128"
Aug 01 00:05:02.386133 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:02.386095    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:11.386657 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:11.386621    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:11.399477 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:11.399450    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:12.300915 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:12.300884    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:34.786460 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:34.786426    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"]
Aug 01 00:05:34.796125 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:05:34.796082    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:05:34.796125 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:05:34.796125 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Aug 01 00:05:34.796125 ip-10-0-140-3 kubenswrapper[2578]:  > containerID="93a00dbfd010d5e9b1cf6cea80a8e13eecc3aa01bb9d561933da7f7a69d97137" cmd=["/bin/sleep","15"]
Aug 01 00:05:34.796390 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:34.796141    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk" message="" err=<
Aug 01 00:05:34.796390 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:05:34.796390 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Aug 01 00:05:34.796390 ip-10-0-140-3 kubenswrapper[2578]:  >
Aug 01 00:05:34.796390 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:05:34.796153    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:05:34.796390 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:05:34.796390 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Aug 01 00:05:34.796390 ip-10-0-140-3 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk" podUID="6103ee63-337a-429f-a3bc-018d7244b50d" containerName="main" containerID="cri-o://93a00dbfd010d5e9b1cf6cea80a8e13eecc3aa01bb9d561933da7f7a69d97137"
Aug 01 00:05:34.796390 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:34.796197    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk" podUID="6103ee63-337a-429f-a3bc-018d7244b50d" containerName="main" containerID="cri-o://93a00dbfd010d5e9b1cf6cea80a8e13eecc3aa01bb9d561933da7f7a69d97137" gracePeriod=60
Aug 01 00:05:35.141531 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.141504    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:35.181235 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.181206    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6103ee63-337a-429f-a3bc-018d7244b50d-tls-certs\") pod \"6103ee63-337a-429f-a3bc-018d7244b50d\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") "
Aug 01 00:05:35.181818 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.181259    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-dshm\") pod \"6103ee63-337a-429f-a3bc-018d7244b50d\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") "
Aug 01 00:05:35.181818 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.181322    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hj5m\" (UniqueName: \"kubernetes.io/projected/6103ee63-337a-429f-a3bc-018d7244b50d-kube-api-access-5hj5m\") pod \"6103ee63-337a-429f-a3bc-018d7244b50d\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") "
Aug 01 00:05:35.181818 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.181370    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-tmp-dir\") pod \"6103ee63-337a-429f-a3bc-018d7244b50d\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") "
Aug 01 00:05:35.181818 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.181393    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-home\") pod \"6103ee63-337a-429f-a3bc-018d7244b50d\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") "
Aug 01 00:05:35.181818 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.181420    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-model-cache\") pod \"6103ee63-337a-429f-a3bc-018d7244b50d\" (UID: \"6103ee63-337a-429f-a3bc-018d7244b50d\") "
Aug 01 00:05:35.181818 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.181808    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6103ee63-337a-429f-a3bc-018d7244b50d" (UID: "6103ee63-337a-429f-a3bc-018d7244b50d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:05:35.182118 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.181827    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-home" (OuterVolumeSpecName: "home") pod "6103ee63-337a-429f-a3bc-018d7244b50d" (UID: "6103ee63-337a-429f-a3bc-018d7244b50d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:05:35.182118 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.181973    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-model-cache" (OuterVolumeSpecName: "model-cache") pod "6103ee63-337a-429f-a3bc-018d7244b50d" (UID: "6103ee63-337a-429f-a3bc-018d7244b50d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:05:35.183998 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.183977    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-dshm" (OuterVolumeSpecName: "dshm") pod "6103ee63-337a-429f-a3bc-018d7244b50d" (UID: "6103ee63-337a-429f-a3bc-018d7244b50d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:05:35.184168 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.184149    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6103ee63-337a-429f-a3bc-018d7244b50d-kube-api-access-5hj5m" (OuterVolumeSpecName: "kube-api-access-5hj5m") pod "6103ee63-337a-429f-a3bc-018d7244b50d" (UID: "6103ee63-337a-429f-a3bc-018d7244b50d"). InnerVolumeSpecName "kube-api-access-5hj5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:05:35.184243 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.184215    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6103ee63-337a-429f-a3bc-018d7244b50d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6103ee63-337a-429f-a3bc-018d7244b50d" (UID: "6103ee63-337a-429f-a3bc-018d7244b50d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:05:35.282574 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.282537    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hj5m\" (UniqueName: \"kubernetes.io/projected/6103ee63-337a-429f-a3bc-018d7244b50d-kube-api-access-5hj5m\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:05:35.282574 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.282567    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-tmp-dir\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:05:35.282574 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.282577    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-home\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:05:35.282851 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.282585    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-model-cache\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:05:35.282851 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.282594    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6103ee63-337a-429f-a3bc-018d7244b50d-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:05:35.282851 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.282601    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6103ee63-337a-429f-a3bc-018d7244b50d-dshm\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:05:35.412367 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.412327    2578 generic.go:356] "Generic (PLEG): container finished" podID="6103ee63-337a-429f-a3bc-018d7244b50d" containerID="93a00dbfd010d5e9b1cf6cea80a8e13eecc3aa01bb9d561933da7f7a69d97137" exitCode=0
Aug 01 00:05:35.412367 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.412365    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"
Aug 01 00:05:35.412632 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.412410    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk" event={"ID":"6103ee63-337a-429f-a3bc-018d7244b50d","Type":"ContainerDied","Data":"93a00dbfd010d5e9b1cf6cea80a8e13eecc3aa01bb9d561933da7f7a69d97137"}
Aug 01 00:05:35.412632 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.412469    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk" event={"ID":"6103ee63-337a-429f-a3bc-018d7244b50d","Type":"ContainerDied","Data":"eee87c3bfd9e328b15613b7de5fb4790c8518335caa574149b2442ad69fe7bbe"}
Aug 01 00:05:35.412632 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.412499    2578 scope.go:117] "RemoveContainer" containerID="93a00dbfd010d5e9b1cf6cea80a8e13eecc3aa01bb9d561933da7f7a69d97137"
Aug 01 00:05:35.425918 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.425900    2578 scope.go:117] "RemoveContainer" containerID="93a00dbfd010d5e9b1cf6cea80a8e13eecc3aa01bb9d561933da7f7a69d97137"
Aug 01 00:05:35.426188 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:05:35.426165    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93a00dbfd010d5e9b1cf6cea80a8e13eecc3aa01bb9d561933da7f7a69d97137\": container with ID starting with 93a00dbfd010d5e9b1cf6cea80a8e13eecc3aa01bb9d561933da7f7a69d97137 not found: ID does not exist" containerID="93a00dbfd010d5e9b1cf6cea80a8e13eecc3aa01bb9d561933da7f7a69d97137"
Aug 01 00:05:35.426229 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.426193    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93a00dbfd010d5e9b1cf6cea80a8e13eecc3aa01bb9d561933da7f7a69d97137"} err="failed to get container status \"93a00dbfd010d5e9b1cf6cea80a8e13eecc3aa01bb9d561933da7f7a69d97137\": rpc error: code = NotFound desc = could not find container \"93a00dbfd010d5e9b1cf6cea80a8e13eecc3aa01bb9d561933da7f7a69d97137\": container with ID starting with 93a00dbfd010d5e9b1cf6cea80a8e13eecc3aa01bb9d561933da7f7a69d97137 not found: ID does not exist"
Aug 01 00:05:35.429496 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.429475    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"]
Aug 01 00:05:35.436935 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:35.436914    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-w5gvk"]
Aug 01 00:05:37.185609 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:05:37.185576    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6103ee63-337a-429f-a3bc-018d7244b50d" path="/var/lib/kubelet/pods/6103ee63-337a-429f-a3bc-018d7244b50d/volumes"
Aug 01 00:06:01.208401 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:01.208369    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:06:01.208937 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:01.208548    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:06:06.893611 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:06.893577    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"]
Aug 01 00:06:06.896161 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:06.894032    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6103ee63-337a-429f-a3bc-018d7244b50d" containerName="main"
Aug 01 00:06:06.896161 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:06.894047    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6103ee63-337a-429f-a3bc-018d7244b50d" containerName="main"
Aug 01 00:06:06.896161 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:06.894116    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6103ee63-337a-429f-a3bc-018d7244b50d" containerName="main"
Aug 01 00:06:06.897586 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:06.897563    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:06.899887 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:06.899860    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Aug 01 00:06:06.900010 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:06.899860    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Aug 01 00:06:06.900300 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:06.900272    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-lvsfb\""
Aug 01 00:06:06.900399 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:06.900367    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Aug 01 00:06:06.900399 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:06.900392    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-kpwv2\""
Aug 01 00:06:06.911560 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:06.911386    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"]
Aug 01 00:06:06.958196 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:06.958161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:06.958373 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:06.958233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:06.958373 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:06.958253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z85tx\" (UniqueName: \"kubernetes.io/projected/9338622d-9afc-4380-a225-c4a55e5704c5-kube-api-access-z85tx\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:06.958373 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:06.958287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:06.958373 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:06.958322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:06.958373 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:06.958364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9338622d-9afc-4380-a225-c4a55e5704c5-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:07.059348 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:07.059304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:07.059348 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:07.059350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z85tx\" (UniqueName: \"kubernetes.io/projected/9338622d-9afc-4380-a225-c4a55e5704c5-kube-api-access-z85tx\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:07.059586 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:07.059384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:07.059586 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:07.059408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:07.059586 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:07.059430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9338622d-9afc-4380-a225-c4a55e5704c5-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:07.059586 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:07.059460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:07.059918 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:07.059887    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:07.060045 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:07.059921    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:07.060045 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:07.059935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:07.061733 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:07.061701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:07.062090 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:07.062072    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9338622d-9afc-4380-a225-c4a55e5704c5-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:07.067463 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:07.067445    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z85tx\" (UniqueName: \"kubernetes.io/projected/9338622d-9afc-4380-a225-c4a55e5704c5-kube-api-access-z85tx\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:07.209307 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:07.209224    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:07.348348 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:07.348316    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"]
Aug 01 00:06:07.350298 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:06:07.350273    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9338622d_9afc_4380_a225_c4a55e5704c5.slice/crio-5810b09fcb34168510531e1896dde82bdb77dff9a41ed7589c64f6cbef919600 WatchSource:0}: Error finding container 5810b09fcb34168510531e1896dde82bdb77dff9a41ed7589c64f6cbef919600: Status 404 returned error can't find the container with id 5810b09fcb34168510531e1896dde82bdb77dff9a41ed7589c64f6cbef919600
Aug 01 00:06:07.352366 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:07.352350    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:06:07.588047 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:07.587957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr" event={"ID":"9338622d-9afc-4380-a225-c4a55e5704c5","Type":"ContainerStarted","Data":"5810b09fcb34168510531e1896dde82bdb77dff9a41ed7589c64f6cbef919600"}
Aug 01 00:06:09.601560 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:09.601525    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr" event={"ID":"9338622d-9afc-4380-a225-c4a55e5704c5","Type":"ContainerStarted","Data":"b7d16d262928fd4cd5c2e09ce56dbd05fbde19e07d90519f9f1740e27751c4fb"}
Aug 01 00:06:09.602069 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:09.601750    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:10.608564 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:10.608533    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr" event={"ID":"9338622d-9afc-4380-a225-c4a55e5704c5","Type":"ContainerStarted","Data":"f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767"}
Aug 01 00:06:10.628897 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:10.628844    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr" podStartSLOduration=3.071942111 podStartE2EDuration="4.628825525s" totalImagesPullingTime="1.556883414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:06:10.626873787 +0000 UTC m=+1209.986764830" watchObservedRunningTime="2026-08-01 00:06:10.628825525 +0000 UTC m=+1209.988716565"
Aug 01 00:06:11.209445 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:11.209406    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:17.209480 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:17.209435    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:17.221830 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:17.221807    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:17.657868 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:17.657833    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:18.663824 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:18.663791    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:40.614965 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:40.614920    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"]
Aug 01 00:06:40.629960 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:06:40.629915    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:06:40.629960 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:06:40.629960 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Aug 01 00:06:40.629960 ip-10-0-140-3 kubenswrapper[2578]:  > containerID="f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767" cmd=["/bin/sleep","15"]
Aug 01 00:06:40.630295 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:40.629984    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr" message="" err=<
Aug 01 00:06:40.630295 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:06:40.630295 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Aug 01 00:06:40.630295 ip-10-0-140-3 kubenswrapper[2578]:  >
Aug 01 00:06:40.630295 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:06:40.629997    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:06:40.630295 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:06:40.630295 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Aug 01 00:06:40.630295 ip-10-0-140-3 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr" podUID="9338622d-9afc-4380-a225-c4a55e5704c5" containerName="main" containerID="cri-o://f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767"
Aug 01 00:06:40.630295 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:40.630042    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr" podUID="9338622d-9afc-4380-a225-c4a55e5704c5" containerName="main" containerID="cri-o://f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767" gracePeriod=60
Aug 01 00:06:40.647597 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:40.647560    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr" podUID="9338622d-9afc-4380-a225-c4a55e5704c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8001/health\": dial tcp 10.134.0.54:8001: connect: connection refused"
Aug 01 00:06:40.843414 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:40.843378    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr" podUID="9338622d-9afc-4380-a225-c4a55e5704c5" containerName="llm-d-routing-sidecar" containerID="cri-o://b7d16d262928fd4cd5c2e09ce56dbd05fbde19e07d90519f9f1740e27751c4fb" gracePeriod=60
Aug 01 00:06:41.105038 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.105014    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:41.144387 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.144351    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-dshm\") pod \"9338622d-9afc-4380-a225-c4a55e5704c5\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") "
Aug 01 00:06:41.144534 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.144448    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z85tx\" (UniqueName: \"kubernetes.io/projected/9338622d-9afc-4380-a225-c4a55e5704c5-kube-api-access-z85tx\") pod \"9338622d-9afc-4380-a225-c4a55e5704c5\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") "
Aug 01 00:06:41.144534 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.144475    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9338622d-9afc-4380-a225-c4a55e5704c5-tls-certs\") pod \"9338622d-9afc-4380-a225-c4a55e5704c5\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") "
Aug 01 00:06:41.144534 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.144494    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-tmp-dir\") pod \"9338622d-9afc-4380-a225-c4a55e5704c5\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") "
Aug 01 00:06:41.144534 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.144522    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-model-cache\") pod \"9338622d-9afc-4380-a225-c4a55e5704c5\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") "
Aug 01 00:06:41.144806 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.144568    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-home\") pod \"9338622d-9afc-4380-a225-c4a55e5704c5\" (UID: \"9338622d-9afc-4380-a225-c4a55e5704c5\") "
Aug 01 00:06:41.144806 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.144793    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9338622d-9afc-4380-a225-c4a55e5704c5" (UID: "9338622d-9afc-4380-a225-c4a55e5704c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:06:41.144916 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.144824    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "9338622d-9afc-4380-a225-c4a55e5704c5" (UID: "9338622d-9afc-4380-a225-c4a55e5704c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:06:41.144971 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.144931    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-model-cache\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:06:41.145028 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.144992    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-tmp-dir\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:06:41.145028 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.145008    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-home" (OuterVolumeSpecName: "home") pod "9338622d-9afc-4380-a225-c4a55e5704c5" (UID: "9338622d-9afc-4380-a225-c4a55e5704c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:06:41.146872 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.146836    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9338622d-9afc-4380-a225-c4a55e5704c5-kube-api-access-z85tx" (OuterVolumeSpecName: "kube-api-access-z85tx") pod "9338622d-9afc-4380-a225-c4a55e5704c5" (UID: "9338622d-9afc-4380-a225-c4a55e5704c5"). InnerVolumeSpecName "kube-api-access-z85tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:06:41.146872 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.146837    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9338622d-9afc-4380-a225-c4a55e5704c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9338622d-9afc-4380-a225-c4a55e5704c5" (UID: "9338622d-9afc-4380-a225-c4a55e5704c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:06:41.147053 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.146981    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-dshm" (OuterVolumeSpecName: "dshm") pod "9338622d-9afc-4380-a225-c4a55e5704c5" (UID: "9338622d-9afc-4380-a225-c4a55e5704c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:06:41.245625 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.245541    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-dshm\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:06:41.245625 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.245569    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z85tx\" (UniqueName: \"kubernetes.io/projected/9338622d-9afc-4380-a225-c4a55e5704c5-kube-api-access-z85tx\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:06:41.245625 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.245579    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9338622d-9afc-4380-a225-c4a55e5704c5-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:06:41.245625 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.245587    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9338622d-9afc-4380-a225-c4a55e5704c5-home\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:06:41.783039 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.783004    2578 generic.go:356] "Generic (PLEG): container finished" podID="9338622d-9afc-4380-a225-c4a55e5704c5" containerID="f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767" exitCode=0
Aug 01 00:06:41.783039 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.783031    2578 generic.go:356] "Generic (PLEG): container finished" podID="9338622d-9afc-4380-a225-c4a55e5704c5" containerID="b7d16d262928fd4cd5c2e09ce56dbd05fbde19e07d90519f9f1740e27751c4fb" exitCode=0
Aug 01 00:06:41.783508 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.783078    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"
Aug 01 00:06:41.783508 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.783093    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr" event={"ID":"9338622d-9afc-4380-a225-c4a55e5704c5","Type":"ContainerDied","Data":"f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767"}
Aug 01 00:06:41.783508 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.783185    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr" event={"ID":"9338622d-9afc-4380-a225-c4a55e5704c5","Type":"ContainerDied","Data":"b7d16d262928fd4cd5c2e09ce56dbd05fbde19e07d90519f9f1740e27751c4fb"}
Aug 01 00:06:41.783508 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.783213    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr" event={"ID":"9338622d-9afc-4380-a225-c4a55e5704c5","Type":"ContainerDied","Data":"5810b09fcb34168510531e1896dde82bdb77dff9a41ed7589c64f6cbef919600"}
Aug 01 00:06:41.783508 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.783240    2578 scope.go:117] "RemoveContainer" containerID="f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767"
Aug 01 00:06:41.792598 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.792575    2578 scope.go:117] "RemoveContainer" containerID="b7d16d262928fd4cd5c2e09ce56dbd05fbde19e07d90519f9f1740e27751c4fb"
Aug 01 00:06:41.799969 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.799945    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"]
Aug 01 00:06:41.801757 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.801734    2578 scope.go:117] "RemoveContainer" containerID="f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767"
Aug 01 00:06:41.802091 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:06:41.802069    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767\": container with ID starting with f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767 not found: ID does not exist" containerID="f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767"
Aug 01 00:06:41.802178 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.802098    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767"} err="failed to get container status \"f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767\": rpc error: code = NotFound desc = could not find container \"f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767\": container with ID starting with f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767 not found: ID does not exist"
Aug 01 00:06:41.802178 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.802117    2578 scope.go:117] "RemoveContainer" containerID="b7d16d262928fd4cd5c2e09ce56dbd05fbde19e07d90519f9f1740e27751c4fb"
Aug 01 00:06:41.802415 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:06:41.802395    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d16d262928fd4cd5c2e09ce56dbd05fbde19e07d90519f9f1740e27751c4fb\": container with ID starting with b7d16d262928fd4cd5c2e09ce56dbd05fbde19e07d90519f9f1740e27751c4fb not found: ID does not exist" containerID="b7d16d262928fd4cd5c2e09ce56dbd05fbde19e07d90519f9f1740e27751c4fb"
Aug 01 00:06:41.802459 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.802423    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d16d262928fd4cd5c2e09ce56dbd05fbde19e07d90519f9f1740e27751c4fb"} err="failed to get container status \"b7d16d262928fd4cd5c2e09ce56dbd05fbde19e07d90519f9f1740e27751c4fb\": rpc error: code = NotFound desc = could not find container \"b7d16d262928fd4cd5c2e09ce56dbd05fbde19e07d90519f9f1740e27751c4fb\": container with ID starting with b7d16d262928fd4cd5c2e09ce56dbd05fbde19e07d90519f9f1740e27751c4fb not found: ID does not exist"
Aug 01 00:06:41.802459 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.802439    2578 scope.go:117] "RemoveContainer" containerID="f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767"
Aug 01 00:06:41.802690 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.802663    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767"} err="failed to get container status \"f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767\": rpc error: code = NotFound desc = could not find container \"f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767\": container with ID starting with f143042abf31a65cbf9ffeb975ccb039c07d259068bf5728dd05d87b5d3e9767 not found: ID does not exist"
Aug 01 00:06:41.802805 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.802691    2578 scope.go:117] "RemoveContainer" containerID="b7d16d262928fd4cd5c2e09ce56dbd05fbde19e07d90519f9f1740e27751c4fb"
Aug 01 00:06:41.803009 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.802948    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d16d262928fd4cd5c2e09ce56dbd05fbde19e07d90519f9f1740e27751c4fb"} err="failed to get container status \"b7d16d262928fd4cd5c2e09ce56dbd05fbde19e07d90519f9f1740e27751c4fb\": rpc error: code = NotFound desc = could not find container \"b7d16d262928fd4cd5c2e09ce56dbd05fbde19e07d90519f9f1740e27751c4fb\": container with ID starting with b7d16d262928fd4cd5c2e09ce56dbd05fbde19e07d90519f9f1740e27751c4fb not found: ID does not exist"
Aug 01 00:06:41.803678 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:41.803661    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-6cbrr"]
Aug 01 00:06:43.185378 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:06:43.185347    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9338622d-9afc-4380-a225-c4a55e5704c5" path="/var/lib/kubelet/pods/9338622d-9afc-4380-a225-c4a55e5704c5/volumes"
Aug 01 00:07:35.396157 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.396121    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"]
Aug 01 00:07:35.396801 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.396777    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9338622d-9afc-4380-a225-c4a55e5704c5" containerName="llm-d-routing-sidecar"
Aug 01 00:07:35.396893 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.396804    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9338622d-9afc-4380-a225-c4a55e5704c5" containerName="llm-d-routing-sidecar"
Aug 01 00:07:35.396893 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.396817    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9338622d-9afc-4380-a225-c4a55e5704c5" containerName="main"
Aug 01 00:07:35.396893 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.396826    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9338622d-9afc-4380-a225-c4a55e5704c5" containerName="main"
Aug 01 00:07:35.397061 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.396949    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9338622d-9afc-4380-a225-c4a55e5704c5" containerName="main"
Aug 01 00:07:35.397061 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.396967    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9338622d-9afc-4380-a225-c4a55e5704c5" containerName="llm-d-routing-sidecar"
Aug 01 00:07:35.400297 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.400273    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.402796 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.402763    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-885f7\""
Aug 01 00:07:35.402929 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.402766    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Aug 01 00:07:35.402929 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.402808    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Aug 01 00:07:35.402929 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.402765    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-kqv2f\""
Aug 01 00:07:35.402929 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.402895    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Aug 01 00:07:35.412790 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.412763    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"]
Aug 01 00:07:35.537540 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.537503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.537755 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.537541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.537755 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.537601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.537755 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.537643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d146f553-fee9-4a5c-9060-86537707a332-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.537755 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.537702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.537939 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.537769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvr77\" (UniqueName: \"kubernetes.io/projected/d146f553-fee9-4a5c-9060-86537707a332-kube-api-access-mvr77\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.638627 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.638587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.638844 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.638641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.638844 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.638667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.638844 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.638702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d146f553-fee9-4a5c-9060-86537707a332-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.638844 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.638767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.638844 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.638803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvr77\" (UniqueName: \"kubernetes.io/projected/d146f553-fee9-4a5c-9060-86537707a332-kube-api-access-mvr77\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.639177 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.639148    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.639177 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.639159    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.639292 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.639268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.641962 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.641939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.642545 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.642522    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d146f553-fee9-4a5c-9060-86537707a332-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.643023 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.643003    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf"]
Aug 01 00:07:35.647013 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.646954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvr77\" (UniqueName: \"kubernetes.io/projected/d146f553-fee9-4a5c-9060-86537707a332-kube-api-access-mvr77\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.647229 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.647214    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf"
Aug 01 00:07:35.648910 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.648892    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-kcv9r\""
Aug 01 00:07:35.654667 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.654648    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf"]
Aug 01 00:07:35.710338 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.710300    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:35.739576 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.739545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c295c7-619a-4565-9535-07d04242d532-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf\" (UID: \"f3c295c7-619a-4565-9535-07d04242d532\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf"
Aug 01 00:07:35.739737 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.739585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w74z4\" (UniqueName: \"kubernetes.io/projected/f3c295c7-619a-4565-9535-07d04242d532-kube-api-access-w74z4\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf\" (UID: \"f3c295c7-619a-4565-9535-07d04242d532\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf"
Aug 01 00:07:35.840411 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.840382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c295c7-619a-4565-9535-07d04242d532-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf\" (UID: \"f3c295c7-619a-4565-9535-07d04242d532\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf"
Aug 01 00:07:35.840570 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.840416    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w74z4\" (UniqueName: \"kubernetes.io/projected/f3c295c7-619a-4565-9535-07d04242d532-kube-api-access-w74z4\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf\" (UID: \"f3c295c7-619a-4565-9535-07d04242d532\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf"
Aug 01 00:07:35.843006 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.842977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c295c7-619a-4565-9535-07d04242d532-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf\" (UID: \"f3c295c7-619a-4565-9535-07d04242d532\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf"
Aug 01 00:07:35.847759 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.847734    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w74z4\" (UniqueName: \"kubernetes.io/projected/f3c295c7-619a-4565-9535-07d04242d532-kube-api-access-w74z4\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf\" (UID: \"f3c295c7-619a-4565-9535-07d04242d532\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf"
Aug 01 00:07:35.857062 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.856900    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"]
Aug 01 00:07:35.859590 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:07:35.859569    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd146f553_fee9_4a5c_9060_86537707a332.slice/crio-0e5fbaa4187c8e11c96332b72fe27886fc37c3916aed0d020d3639c4fb479009 WatchSource:0}: Error finding container 0e5fbaa4187c8e11c96332b72fe27886fc37c3916aed0d020d3639c4fb479009: Status 404 returned error can't find the container with id 0e5fbaa4187c8e11c96332b72fe27886fc37c3916aed0d020d3639c4fb479009
Aug 01 00:07:35.970802 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:35.970767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf"
Aug 01 00:07:36.076062 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:36.076028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4" event={"ID":"d146f553-fee9-4a5c-9060-86537707a332","Type":"ContainerStarted","Data":"1096833a5c2606b37f433097ae5f93da6787b661fe7c15d7466b57b3ec3285b8"}
Aug 01 00:07:36.076252 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:36.076082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4" event={"ID":"d146f553-fee9-4a5c-9060-86537707a332","Type":"ContainerStarted","Data":"0e5fbaa4187c8e11c96332b72fe27886fc37c3916aed0d020d3639c4fb479009"}
Aug 01 00:07:36.076252 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:36.076110    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:36.114528 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:36.114505    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf"]
Aug 01 00:07:36.116161 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:07:36.116138    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c295c7_619a_4565_9535_07d04242d532.slice/crio-a8bf1bdf451b17c11a83c271bec3225231b1511c988481360216e00b70dea307 WatchSource:0}: Error finding container a8bf1bdf451b17c11a83c271bec3225231b1511c988481360216e00b70dea307: Status 404 returned error can't find the container with id a8bf1bdf451b17c11a83c271bec3225231b1511c988481360216e00b70dea307
Aug 01 00:07:37.083803 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:37.083771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4" event={"ID":"d146f553-fee9-4a5c-9060-86537707a332","Type":"ContainerStarted","Data":"1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b"}
Aug 01 00:07:37.085357 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:37.085330    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf" event={"ID":"f3c295c7-619a-4565-9535-07d04242d532","Type":"ContainerStarted","Data":"164ab07fe92e21f339e147cdd4fe43752e65a7d903f0deb89474b3f21348c780"}
Aug 01 00:07:37.085459 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:37.085376    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf"
Aug 01 00:07:37.085459 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:37.085393    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf" event={"ID":"f3c295c7-619a-4565-9535-07d04242d532","Type":"ContainerStarted","Data":"a8bf1bdf451b17c11a83c271bec3225231b1511c988481360216e00b70dea307"}
Aug 01 00:07:37.103438 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:37.103383    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4" podStartSLOduration=2.103366312 podStartE2EDuration="2.103366312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:07:37.102117623 +0000 UTC m=+1296.462008664" watchObservedRunningTime="2026-08-01 00:07:37.103366312 +0000 UTC m=+1296.463257354"
Aug 01 00:07:37.120774 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:37.120701    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf" podStartSLOduration=2.120685812 podStartE2EDuration="2.120685812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:07:37.118401443 +0000 UTC m=+1296.478292495" watchObservedRunningTime="2026-08-01 00:07:37.120685812 +0000 UTC m=+1296.480576853"
Aug 01 00:07:37.710822 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:37.710783    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:45.710854 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:45.710802    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:45.725165 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:45.725130    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:45.729137 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:45.729116    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:07:46.151579 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:07:46.151546    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:08:08.093000 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:08.092970    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf"
Aug 01 00:08:09.293913 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.293866    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf"]
Aug 01 00:08:09.296914 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.296879    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"]
Aug 01 00:08:09.307534 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:08:09.307493    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:08:09.307534 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:08:09.307534 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Aug 01 00:08:09.307534 ip-10-0-140-3 kubenswrapper[2578]:  > containerID="1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b" cmd=["/bin/sleep","15"]
Aug 01 00:08:09.307863 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.307553    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4" message="" err=<
Aug 01 00:08:09.307863 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:08:09.307863 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Aug 01 00:08:09.307863 ip-10-0-140-3 kubenswrapper[2578]:  >
Aug 01 00:08:09.307863 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:08:09.307570    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:08:09.307863 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:08:09.307863 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Aug 01 00:08:09.307863 ip-10-0-140-3 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4" podUID="d146f553-fee9-4a5c-9060-86537707a332" containerName="main" containerID="cri-o://1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b"
Aug 01 00:08:09.307863 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.307615    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4" podUID="d146f553-fee9-4a5c-9060-86537707a332" containerName="main" containerID="cri-o://1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b" gracePeriod=60
Aug 01 00:08:09.527781 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.527746    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4" podUID="d146f553-fee9-4a5c-9060-86537707a332" containerName="llm-d-routing-sidecar" containerID="cri-o://1096833a5c2606b37f433097ae5f93da6787b661fe7c15d7466b57b3ec3285b8" gracePeriod=60
Aug 01 00:08:09.791785 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.791754    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:08:09.833358 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.833324    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-home\") pod \"d146f553-fee9-4a5c-9060-86537707a332\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") "
Aug 01 00:08:09.833552 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.833384    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-model-cache\") pod \"d146f553-fee9-4a5c-9060-86537707a332\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") "
Aug 01 00:08:09.833552 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.833434    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d146f553-fee9-4a5c-9060-86537707a332-tls-certs\") pod \"d146f553-fee9-4a5c-9060-86537707a332\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") "
Aug 01 00:08:09.833552 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.833529    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvr77\" (UniqueName: \"kubernetes.io/projected/d146f553-fee9-4a5c-9060-86537707a332-kube-api-access-mvr77\") pod \"d146f553-fee9-4a5c-9060-86537707a332\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") "
Aug 01 00:08:09.833756 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.833556    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-dshm\") pod \"d146f553-fee9-4a5c-9060-86537707a332\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") "
Aug 01 00:08:09.833827 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.833747    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-model-cache" (OuterVolumeSpecName: "model-cache") pod "d146f553-fee9-4a5c-9060-86537707a332" (UID: "d146f553-fee9-4a5c-9060-86537707a332"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:08:09.833827 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.833759    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-tmp-dir\") pod \"d146f553-fee9-4a5c-9060-86537707a332\" (UID: \"d146f553-fee9-4a5c-9060-86537707a332\") "
Aug 01 00:08:09.834102 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.834072    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-model-cache\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:08:09.834230 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.834100    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-home" (OuterVolumeSpecName: "home") pod "d146f553-fee9-4a5c-9060-86537707a332" (UID: "d146f553-fee9-4a5c-9060-86537707a332"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:08:09.834230 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.834119    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d146f553-fee9-4a5c-9060-86537707a332" (UID: "d146f553-fee9-4a5c-9060-86537707a332"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:08:09.836032 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.836012    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d146f553-fee9-4a5c-9060-86537707a332-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d146f553-fee9-4a5c-9060-86537707a332" (UID: "d146f553-fee9-4a5c-9060-86537707a332"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:08:09.836430 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.836401    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-dshm" (OuterVolumeSpecName: "dshm") pod "d146f553-fee9-4a5c-9060-86537707a332" (UID: "d146f553-fee9-4a5c-9060-86537707a332"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:08:09.836516 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.836426    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d146f553-fee9-4a5c-9060-86537707a332-kube-api-access-mvr77" (OuterVolumeSpecName: "kube-api-access-mvr77") pod "d146f553-fee9-4a5c-9060-86537707a332" (UID: "d146f553-fee9-4a5c-9060-86537707a332"). InnerVolumeSpecName "kube-api-access-mvr77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:08:09.935395 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.935302    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-home\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:08:09.935395 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.935337    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d146f553-fee9-4a5c-9060-86537707a332-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:08:09.935395 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.935346    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvr77\" (UniqueName: \"kubernetes.io/projected/d146f553-fee9-4a5c-9060-86537707a332-kube-api-access-mvr77\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:08:09.935395 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.935356    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-dshm\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:08:09.935395 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:09.935364    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d146f553-fee9-4a5c-9060-86537707a332-tmp-dir\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:08:10.278968 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:10.278866    2578 generic.go:356] "Generic (PLEG): container finished" podID="d146f553-fee9-4a5c-9060-86537707a332" containerID="1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b" exitCode=0
Aug 01 00:08:10.278968 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:10.278890    2578 generic.go:356] "Generic (PLEG): container finished" podID="d146f553-fee9-4a5c-9060-86537707a332" containerID="1096833a5c2606b37f433097ae5f93da6787b661fe7c15d7466b57b3ec3285b8" exitCode=0
Aug 01 00:08:10.278968 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:10.278940    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"
Aug 01 00:08:10.279253 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:10.278961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4" event={"ID":"d146f553-fee9-4a5c-9060-86537707a332","Type":"ContainerDied","Data":"1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b"}
Aug 01 00:08:10.279253 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:10.279051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4" event={"ID":"d146f553-fee9-4a5c-9060-86537707a332","Type":"ContainerDied","Data":"1096833a5c2606b37f433097ae5f93da6787b661fe7c15d7466b57b3ec3285b8"}
Aug 01 00:08:10.279253 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:10.279075    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4" event={"ID":"d146f553-fee9-4a5c-9060-86537707a332","Type":"ContainerDied","Data":"0e5fbaa4187c8e11c96332b72fe27886fc37c3916aed0d020d3639c4fb479009"}
Aug 01 00:08:10.279253 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:10.279102    2578 scope.go:117] "RemoveContainer" containerID="1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b"
Aug 01 00:08:10.288790 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:10.288767    2578 scope.go:117] "RemoveContainer" containerID="1096833a5c2606b37f433097ae5f93da6787b661fe7c15d7466b57b3ec3285b8"
Aug 01 00:08:10.297569 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:10.297545    2578 scope.go:117] "RemoveContainer" containerID="1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b"
Aug 01 00:08:10.298289 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:08:10.298263    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b\": container with ID starting with 1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b not found: ID does not exist" containerID="1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b"
Aug 01 00:08:10.298385 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:10.298294    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b"} err="failed to get container status \"1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b\": rpc error: code = NotFound desc = could not find container \"1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b\": container with ID starting with 1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b not found: ID does not exist"
Aug 01 00:08:10.298385 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:10.298313    2578 scope.go:117] "RemoveContainer" containerID="1096833a5c2606b37f433097ae5f93da6787b661fe7c15d7466b57b3ec3285b8"
Aug 01 00:08:10.298614 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:08:10.298590    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1096833a5c2606b37f433097ae5f93da6787b661fe7c15d7466b57b3ec3285b8\": container with ID starting with 1096833a5c2606b37f433097ae5f93da6787b661fe7c15d7466b57b3ec3285b8 not found: ID does not exist" containerID="1096833a5c2606b37f433097ae5f93da6787b661fe7c15d7466b57b3ec3285b8"
Aug 01 00:08:10.298696 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:10.298620    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1096833a5c2606b37f433097ae5f93da6787b661fe7c15d7466b57b3ec3285b8"} err="failed to get container status \"1096833a5c2606b37f433097ae5f93da6787b661fe7c15d7466b57b3ec3285b8\": rpc error: code = NotFound desc = could not find container \"1096833a5c2606b37f433097ae5f93da6787b661fe7c15d7466b57b3ec3285b8\": container with ID starting with 1096833a5c2606b37f433097ae5f93da6787b661fe7c15d7466b57b3ec3285b8 not found: ID does not exist"
Aug 01 00:08:10.298696 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:10.298638    2578 scope.go:117] "RemoveContainer" containerID="1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b"
Aug 01 00:08:10.298945 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:10.298925    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b"} err="failed to get container status \"1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b\": rpc error: code = NotFound desc = could not find container \"1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b\": container with ID starting with 1ce29df3a68bf77d95c281e7e3ff798a8bcda077a311a817c7a12fbde85d6d0b not found: ID does not exist"
Aug 01 00:08:10.299013 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:10.298946    2578 scope.go:117] "RemoveContainer" containerID="1096833a5c2606b37f433097ae5f93da6787b661fe7c15d7466b57b3ec3285b8"
Aug 01 00:08:10.299162 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:10.299142    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1096833a5c2606b37f433097ae5f93da6787b661fe7c15d7466b57b3ec3285b8"} err="failed to get container status \"1096833a5c2606b37f433097ae5f93da6787b661fe7c15d7466b57b3ec3285b8\": rpc error: code = NotFound desc = could not find container \"1096833a5c2606b37f433097ae5f93da6787b661fe7c15d7466b57b3ec3285b8\": container with ID starting with 1096833a5c2606b37f433097ae5f93da6787b661fe7c15d7466b57b3ec3285b8 not found: ID does not exist"
Aug 01 00:08:10.300221 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:10.300203    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"]
Aug 01 00:08:10.305440 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:10.305418    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-2dgc4"]
Aug 01 00:08:11.185580 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:11.185546    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d146f553-fee9-4a5c-9060-86537707a332" path="/var/lib/kubelet/pods/d146f553-fee9-4a5c-9060-86537707a332/volumes"
Aug 01 00:08:18.092705 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:18.092663    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf" podUID="f3c295c7-619a-4565-9535-07d04242d532" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:08:24.317985 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:24.317900    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf" podUID="f3c295c7-619a-4565-9535-07d04242d532" containerName="main" containerID="cri-o://164ab07fe92e21f339e147cdd4fe43752e65a7d903f0deb89474b3f21348c780" gracePeriod=45
Aug 01 00:08:24.586211 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:24.586182    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf"
Aug 01 00:08:24.669694 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:24.669656    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c295c7-619a-4565-9535-07d04242d532-tls-certs\") pod \"f3c295c7-619a-4565-9535-07d04242d532\" (UID: \"f3c295c7-619a-4565-9535-07d04242d532\") "
Aug 01 00:08:24.669909 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:24.669710    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w74z4\" (UniqueName: \"kubernetes.io/projected/f3c295c7-619a-4565-9535-07d04242d532-kube-api-access-w74z4\") pod \"f3c295c7-619a-4565-9535-07d04242d532\" (UID: \"f3c295c7-619a-4565-9535-07d04242d532\") "
Aug 01 00:08:24.672060 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:24.672022    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3c295c7-619a-4565-9535-07d04242d532-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3c295c7-619a-4565-9535-07d04242d532" (UID: "f3c295c7-619a-4565-9535-07d04242d532"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:08:24.672060 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:24.672053    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c295c7-619a-4565-9535-07d04242d532-kube-api-access-w74z4" (OuterVolumeSpecName: "kube-api-access-w74z4") pod "f3c295c7-619a-4565-9535-07d04242d532" (UID: "f3c295c7-619a-4565-9535-07d04242d532"). InnerVolumeSpecName "kube-api-access-w74z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:08:24.770659 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:24.770616    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c295c7-619a-4565-9535-07d04242d532-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:08:24.770659 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:24.770653    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w74z4\" (UniqueName: \"kubernetes.io/projected/f3c295c7-619a-4565-9535-07d04242d532-kube-api-access-w74z4\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:08:25.368978 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:25.368944    2578 generic.go:356] "Generic (PLEG): container finished" podID="f3c295c7-619a-4565-9535-07d04242d532" containerID="164ab07fe92e21f339e147cdd4fe43752e65a7d903f0deb89474b3f21348c780" exitCode=0
Aug 01 00:08:25.369480 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:25.368985    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf"
Aug 01 00:08:25.369480 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:25.369028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf" event={"ID":"f3c295c7-619a-4565-9535-07d04242d532","Type":"ContainerDied","Data":"164ab07fe92e21f339e147cdd4fe43752e65a7d903f0deb89474b3f21348c780"}
Aug 01 00:08:25.369480 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:25.369092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf" event={"ID":"f3c295c7-619a-4565-9535-07d04242d532","Type":"ContainerDied","Data":"a8bf1bdf451b17c11a83c271bec3225231b1511c988481360216e00b70dea307"}
Aug 01 00:08:25.369480 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:25.369124    2578 scope.go:117] "RemoveContainer" containerID="164ab07fe92e21f339e147cdd4fe43752e65a7d903f0deb89474b3f21348c780"
Aug 01 00:08:25.377880 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:25.377859    2578 scope.go:117] "RemoveContainer" containerID="164ab07fe92e21f339e147cdd4fe43752e65a7d903f0deb89474b3f21348c780"
Aug 01 00:08:25.378180 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:08:25.378155    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"164ab07fe92e21f339e147cdd4fe43752e65a7d903f0deb89474b3f21348c780\": container with ID starting with 164ab07fe92e21f339e147cdd4fe43752e65a7d903f0deb89474b3f21348c780 not found: ID does not exist" containerID="164ab07fe92e21f339e147cdd4fe43752e65a7d903f0deb89474b3f21348c780"
Aug 01 00:08:25.378241 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:25.378179    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"164ab07fe92e21f339e147cdd4fe43752e65a7d903f0deb89474b3f21348c780"} err="failed to get container status \"164ab07fe92e21f339e147cdd4fe43752e65a7d903f0deb89474b3f21348c780\": rpc error: code = NotFound desc = could not find container \"164ab07fe92e21f339e147cdd4fe43752e65a7d903f0deb89474b3f21348c780\": container with ID starting with 164ab07fe92e21f339e147cdd4fe43752e65a7d903f0deb89474b3f21348c780 not found: ID does not exist"
Aug 01 00:08:25.384490 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:25.384460    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf"]
Aug 01 00:08:25.387462 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:25.387440    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqdqsf"]
Aug 01 00:08:27.186124 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:27.186089    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3c295c7-619a-4565-9535-07d04242d532" path="/var/lib/kubelet/pods/f3c295c7-619a-4565-9535-07d04242d532/volumes"
Aug 01 00:08:44.382174 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.382140    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"]
Aug 01 00:08:44.382772 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.382749    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d146f553-fee9-4a5c-9060-86537707a332" containerName="main"
Aug 01 00:08:44.382877 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.382776    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d146f553-fee9-4a5c-9060-86537707a332" containerName="main"
Aug 01 00:08:44.382877 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.382809    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d146f553-fee9-4a5c-9060-86537707a332" containerName="llm-d-routing-sidecar"
Aug 01 00:08:44.382877 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.382818    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d146f553-fee9-4a5c-9060-86537707a332" containerName="llm-d-routing-sidecar"
Aug 01 00:08:44.382877 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.382828    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c295c7-619a-4565-9535-07d04242d532" containerName="main"
Aug 01 00:08:44.382877 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.382837    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c295c7-619a-4565-9535-07d04242d532" containerName="main"
Aug 01 00:08:44.383154 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.382928    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c295c7-619a-4565-9535-07d04242d532" containerName="main"
Aug 01 00:08:44.383154 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.382939    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d146f553-fee9-4a5c-9060-86537707a332" containerName="llm-d-routing-sidecar"
Aug 01 00:08:44.383154 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.382951    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d146f553-fee9-4a5c-9060-86537707a332" containerName="main"
Aug 01 00:08:44.386256 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.386235    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.388112 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.388089    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:08:44.388228 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.388089    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Aug 01 00:08:44.388228 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.388142    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-9gzgn\""
Aug 01 00:08:44.388228 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.388089    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Aug 01 00:08:44.388228 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.388131    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Aug 01 00:08:44.388579 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.388563    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Aug 01 00:08:44.395008 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.394988    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"]
Aug 01 00:08:44.537970 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.537933    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vzg2\" (UniqueName: \"kubernetes.io/projected/8d6082ce-ee02-467d-8795-57f5f0b7ef06-kube-api-access-7vzg2\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.538176 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.538006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-dshm\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.538176 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.538071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-model-cache\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.538176 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.538093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d6082ce-ee02-467d-8795-57f5f0b7ef06-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.538176 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.538138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-home\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.538381 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.538175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.538381 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.538204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6082ce-ee02-467d-8795-57f5f0b7ef06-tls-certs\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.538381 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.538238    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.619944 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.619909    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd"]
Aug 01 00:08:44.623690 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.623667    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd"
Aug 01 00:08:44.625854 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.625826    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-lqxvx\""
Aug 01 00:08:44.635273 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.635196    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd"]
Aug 01 00:08:44.639129 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.639105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-dshm\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.639243 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.639163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-model-cache\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.639243 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.639192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d6082ce-ee02-467d-8795-57f5f0b7ef06-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.639243 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.639217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-home\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.639243 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.639232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.639440 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.639247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6082ce-ee02-467d-8795-57f5f0b7ef06-tls-certs\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.639440 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.639272    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.639440 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.639315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vzg2\" (UniqueName: \"kubernetes.io/projected/8d6082ce-ee02-467d-8795-57f5f0b7ef06-kube-api-access-7vzg2\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.639619 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.639596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-model-cache\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.639708 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.639683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-home\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.640085 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.640064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.640214 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.640159    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d6082ce-ee02-467d-8795-57f5f0b7ef06-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.640333 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.640260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.642292 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.642270    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-dshm\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.642419 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.642336    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6082ce-ee02-467d-8795-57f5f0b7ef06-tls-certs\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.646992 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.646973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vzg2\" (UniqueName: \"kubernetes.io/projected/8d6082ce-ee02-467d-8795-57f5f0b7ef06-kube-api-access-7vzg2\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2mj8r\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.698948 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.698910    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:44.740159 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.740129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfmtr\" (UniqueName: \"kubernetes.io/projected/18061ebd-edcd-436f-bb27-5aa163e88ce9-kube-api-access-gfmtr\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd\" (UID: \"18061ebd-edcd-436f-bb27-5aa163e88ce9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd"
Aug 01 00:08:44.740330 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.740293    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18061ebd-edcd-436f-bb27-5aa163e88ce9-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd\" (UID: \"18061ebd-edcd-436f-bb27-5aa163e88ce9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd"
Aug 01 00:08:44.841286 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.841250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18061ebd-edcd-436f-bb27-5aa163e88ce9-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd\" (UID: \"18061ebd-edcd-436f-bb27-5aa163e88ce9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd"
Aug 01 00:08:44.841453 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.841311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfmtr\" (UniqueName: \"kubernetes.io/projected/18061ebd-edcd-436f-bb27-5aa163e88ce9-kube-api-access-gfmtr\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd\" (UID: \"18061ebd-edcd-436f-bb27-5aa163e88ce9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd"
Aug 01 00:08:44.844839 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.844804    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18061ebd-edcd-436f-bb27-5aa163e88ce9-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd\" (UID: \"18061ebd-edcd-436f-bb27-5aa163e88ce9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd"
Aug 01 00:08:44.850755 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.850113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfmtr\" (UniqueName: \"kubernetes.io/projected/18061ebd-edcd-436f-bb27-5aa163e88ce9-kube-api-access-gfmtr\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd\" (UID: \"18061ebd-edcd-436f-bb27-5aa163e88ce9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd"
Aug 01 00:08:44.850891 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.850856    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"]
Aug 01 00:08:44.853036 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:08:44.853004    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6082ce_ee02_467d_8795_57f5f0b7ef06.slice/crio-9db389627bb37676c209cd7ce7d0f823dc1d704cc7704ea887d653a23fa3f70f WatchSource:0}: Error finding container 9db389627bb37676c209cd7ce7d0f823dc1d704cc7704ea887d653a23fa3f70f: Status 404 returned error can't find the container with id 9db389627bb37676c209cd7ce7d0f823dc1d704cc7704ea887d653a23fa3f70f
Aug 01 00:08:44.936903 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:44.936866    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd"
Aug 01 00:08:45.088697 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:45.088664    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd"]
Aug 01 00:08:45.091976 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:08:45.091943    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18061ebd_edcd_436f_bb27_5aa163e88ce9.slice/crio-018f7b118d0d1de908225372e50d8e76b27a0855cb58bde70f5ee8a7d8115955 WatchSource:0}: Error finding container 018f7b118d0d1de908225372e50d8e76b27a0855cb58bde70f5ee8a7d8115955: Status 404 returned error can't find the container with id 018f7b118d0d1de908225372e50d8e76b27a0855cb58bde70f5ee8a7d8115955
Aug 01 00:08:45.478260 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:45.478225    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd" event={"ID":"18061ebd-edcd-436f-bb27-5aa163e88ce9","Type":"ContainerStarted","Data":"9a4ad676478bd6a1337168747304052acb4af3c3baa1c0c7274f203c2d590a74"}
Aug 01 00:08:45.478795 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:45.478288    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd" event={"ID":"18061ebd-edcd-436f-bb27-5aa163e88ce9","Type":"ContainerStarted","Data":"018f7b118d0d1de908225372e50d8e76b27a0855cb58bde70f5ee8a7d8115955"}
Aug 01 00:08:45.478795 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:45.478503    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd"
Aug 01 00:08:45.480064 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:45.480038    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r" event={"ID":"8d6082ce-ee02-467d-8795-57f5f0b7ef06","Type":"ContainerStarted","Data":"9db389627bb37676c209cd7ce7d0f823dc1d704cc7704ea887d653a23fa3f70f"}
Aug 01 00:08:45.495578 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:45.495519    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd" podStartSLOduration=1.495498798 podStartE2EDuration="1.495498798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:08:45.492909681 +0000 UTC m=+1364.852800743" watchObservedRunningTime="2026-08-01 00:08:45.495498798 +0000 UTC m=+1364.855389819"
Aug 01 00:08:49.505612 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:49.505568    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r" event={"ID":"8d6082ce-ee02-467d-8795-57f5f0b7ef06","Type":"ContainerStarted","Data":"2ad33abda3edcc56ad57ee12c467a97ca41e329283e3d2c50661097baa24256f"}
Aug 01 00:08:51.520187 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:51.520150    2578 generic.go:356] "Generic (PLEG): container finished" podID="8d6082ce-ee02-467d-8795-57f5f0b7ef06" containerID="2ad33abda3edcc56ad57ee12c467a97ca41e329283e3d2c50661097baa24256f" exitCode=0
Aug 01 00:08:51.520680 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:51.520232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r" event={"ID":"8d6082ce-ee02-467d-8795-57f5f0b7ef06","Type":"ContainerDied","Data":"2ad33abda3edcc56ad57ee12c467a97ca41e329283e3d2c50661097baa24256f"}
Aug 01 00:08:52.528940 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:52.528910    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r" event={"ID":"8d6082ce-ee02-467d-8795-57f5f0b7ef06","Type":"ContainerStarted","Data":"2e3fa20d0a9f97019964219e4fa8ecbe4fdb3308ab9afaff3251ca8c9bcd4ac3"}
Aug 01 00:08:52.550485 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:52.550428    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r" podStartSLOduration=2.534066017 podStartE2EDuration="8.550409863s" totalImagesPullingTime="3.813974564s" totalInitContainerRuntime="2.202369282s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:08:52.549772109 +0000 UTC m=+1371.909663178" watchObservedRunningTime="2026-08-01 00:08:52.550409863 +0000 UTC m=+1371.910300905"
Aug 01 00:08:52.699608 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:52.699550    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:54.699545 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:54.699504    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:54.712227 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:54.712201    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:08:55.555926 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:08:55.555894    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:09:16.488934 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:09:16.488903    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd"
Aug 01 00:10:13.785829 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:13.785746    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"]
Aug 01 00:10:13.789127 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:13.789086    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd"]
Aug 01 00:10:13.798556 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:10:13.798520    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:10:13.798556 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:10:13.798556 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Aug 01 00:10:13.798556 ip-10-0-140-3 kubenswrapper[2578]:  > containerID="2e3fa20d0a9f97019964219e4fa8ecbe4fdb3308ab9afaff3251ca8c9bcd4ac3" cmd=["/bin/sleep","15"]
Aug 01 00:10:13.798895 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:13.798575    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r" message="" err=<
Aug 01 00:10:13.798895 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:10:13.798895 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Aug 01 00:10:13.798895 ip-10-0-140-3 kubenswrapper[2578]:  >
Aug 01 00:10:13.798895 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:10:13.798592    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:10:13.798895 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:10:13.798895 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Aug 01 00:10:13.798895 ip-10-0-140-3 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r" podUID="8d6082ce-ee02-467d-8795-57f5f0b7ef06" containerName="main" containerID="cri-o://2e3fa20d0a9f97019964219e4fa8ecbe4fdb3308ab9afaff3251ca8c9bcd4ac3"
Aug 01 00:10:13.798895 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:13.798635    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r" podUID="8d6082ce-ee02-467d-8795-57f5f0b7ef06" containerName="main" containerID="cri-o://2e3fa20d0a9f97019964219e4fa8ecbe4fdb3308ab9afaff3251ca8c9bcd4ac3" gracePeriod=60
Aug 01 00:10:14.165032 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.165005    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:10:14.260293 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.260263    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-model-cache\") pod \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") "
Aug 01 00:10:14.260293 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.260297    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-home\") pod \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") "
Aug 01 00:10:14.260520 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.260344    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6082ce-ee02-467d-8795-57f5f0b7ef06-tls-certs\") pod \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") "
Aug 01 00:10:14.260520 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.260376    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d6082ce-ee02-467d-8795-57f5f0b7ef06-cabundle-cert\") pod \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") "
Aug 01 00:10:14.260520 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.260406    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-dshm\") pod \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") "
Aug 01 00:10:14.260520 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.260503    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vzg2\" (UniqueName: \"kubernetes.io/projected/8d6082ce-ee02-467d-8795-57f5f0b7ef06-kube-api-access-7vzg2\") pod \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") "
Aug 01 00:10:14.260762 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.260546    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-tmp-dir\") pod \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") "
Aug 01 00:10:14.260762 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.260580    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-kserve-provision-location\") pod \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\" (UID: \"8d6082ce-ee02-467d-8795-57f5f0b7ef06\") "
Aug 01 00:10:14.260762 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.260600    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-home" (OuterVolumeSpecName: "home") pod "8d6082ce-ee02-467d-8795-57f5f0b7ef06" (UID: "8d6082ce-ee02-467d-8795-57f5f0b7ef06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:10:14.260925 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.260788    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d6082ce-ee02-467d-8795-57f5f0b7ef06-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8d6082ce-ee02-467d-8795-57f5f0b7ef06" (UID: "8d6082ce-ee02-467d-8795-57f5f0b7ef06"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:10:14.260925 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.260803    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d6082ce-ee02-467d-8795-57f5f0b7ef06" (UID: "8d6082ce-ee02-467d-8795-57f5f0b7ef06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:10:14.261032 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.260944    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-home\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.261032 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.260963    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d6082ce-ee02-467d-8795-57f5f0b7ef06-cabundle-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.261032 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.260978    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-tmp-dir\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.261135 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.261028    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d6082ce-ee02-467d-8795-57f5f0b7ef06" (UID: "8d6082ce-ee02-467d-8795-57f5f0b7ef06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:10:14.262965 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.262939    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d6082ce-ee02-467d-8795-57f5f0b7ef06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d6082ce-ee02-467d-8795-57f5f0b7ef06" (UID: "8d6082ce-ee02-467d-8795-57f5f0b7ef06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:10:14.263083 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.263001    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-dshm" (OuterVolumeSpecName: "dshm") pod "8d6082ce-ee02-467d-8795-57f5f0b7ef06" (UID: "8d6082ce-ee02-467d-8795-57f5f0b7ef06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:10:14.263083 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.263071    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6082ce-ee02-467d-8795-57f5f0b7ef06-kube-api-access-7vzg2" (OuterVolumeSpecName: "kube-api-access-7vzg2") pod "8d6082ce-ee02-467d-8795-57f5f0b7ef06" (UID: "8d6082ce-ee02-467d-8795-57f5f0b7ef06"). InnerVolumeSpecName "kube-api-access-7vzg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:10:14.281017 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.280990    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d6082ce-ee02-467d-8795-57f5f0b7ef06" (UID: "8d6082ce-ee02-467d-8795-57f5f0b7ef06"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:10:14.362449 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.362347    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vzg2\" (UniqueName: \"kubernetes.io/projected/8d6082ce-ee02-467d-8795-57f5f0b7ef06-kube-api-access-7vzg2\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.362449 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.362379    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-kserve-provision-location\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.362449 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.362395    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-model-cache\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.362449 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.362406    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6082ce-ee02-467d-8795-57f5f0b7ef06-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.362449 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.362416    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6082ce-ee02-467d-8795-57f5f0b7ef06-dshm\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.984973 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.984941    2578 generic.go:356] "Generic (PLEG): container finished" podID="8d6082ce-ee02-467d-8795-57f5f0b7ef06" containerID="2e3fa20d0a9f97019964219e4fa8ecbe4fdb3308ab9afaff3251ca8c9bcd4ac3" exitCode=0
Aug 01 00:10:14.985473 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.985004    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"
Aug 01 00:10:14.985473 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.985034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r" event={"ID":"8d6082ce-ee02-467d-8795-57f5f0b7ef06","Type":"ContainerDied","Data":"2e3fa20d0a9f97019964219e4fa8ecbe4fdb3308ab9afaff3251ca8c9bcd4ac3"}
Aug 01 00:10:14.985473 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.985091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r" event={"ID":"8d6082ce-ee02-467d-8795-57f5f0b7ef06","Type":"ContainerDied","Data":"9db389627bb37676c209cd7ce7d0f823dc1d704cc7704ea887d653a23fa3f70f"}
Aug 01 00:10:14.985473 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:14.985115    2578 scope.go:117] "RemoveContainer" containerID="2e3fa20d0a9f97019964219e4fa8ecbe4fdb3308ab9afaff3251ca8c9bcd4ac3"
Aug 01 00:10:15.002622 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:15.002601    2578 scope.go:117] "RemoveContainer" containerID="2ad33abda3edcc56ad57ee12c467a97ca41e329283e3d2c50661097baa24256f"
Aug 01 00:10:15.010196 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:15.010172    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"]
Aug 01 00:10:15.013401 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:15.013379    2578 scope.go:117] "RemoveContainer" containerID="2e3fa20d0a9f97019964219e4fa8ecbe4fdb3308ab9afaff3251ca8c9bcd4ac3"
Aug 01 00:10:15.013703 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:10:15.013683    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e3fa20d0a9f97019964219e4fa8ecbe4fdb3308ab9afaff3251ca8c9bcd4ac3\": container with ID starting with 2e3fa20d0a9f97019964219e4fa8ecbe4fdb3308ab9afaff3251ca8c9bcd4ac3 not found: ID does not exist" containerID="2e3fa20d0a9f97019964219e4fa8ecbe4fdb3308ab9afaff3251ca8c9bcd4ac3"
Aug 01 00:10:15.013824 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:15.013739    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e3fa20d0a9f97019964219e4fa8ecbe4fdb3308ab9afaff3251ca8c9bcd4ac3"} err="failed to get container status \"2e3fa20d0a9f97019964219e4fa8ecbe4fdb3308ab9afaff3251ca8c9bcd4ac3\": rpc error: code = NotFound desc = could not find container \"2e3fa20d0a9f97019964219e4fa8ecbe4fdb3308ab9afaff3251ca8c9bcd4ac3\": container with ID starting with 2e3fa20d0a9f97019964219e4fa8ecbe4fdb3308ab9afaff3251ca8c9bcd4ac3 not found: ID does not exist"
Aug 01 00:10:15.013824 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:15.013761    2578 scope.go:117] "RemoveContainer" containerID="2ad33abda3edcc56ad57ee12c467a97ca41e329283e3d2c50661097baa24256f"
Aug 01 00:10:15.014044 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:10:15.014022    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad33abda3edcc56ad57ee12c467a97ca41e329283e3d2c50661097baa24256f\": container with ID starting with 2ad33abda3edcc56ad57ee12c467a97ca41e329283e3d2c50661097baa24256f not found: ID does not exist" containerID="2ad33abda3edcc56ad57ee12c467a97ca41e329283e3d2c50661097baa24256f"
Aug 01 00:10:15.014103 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:15.014050    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad33abda3edcc56ad57ee12c467a97ca41e329283e3d2c50661097baa24256f"} err="failed to get container status \"2ad33abda3edcc56ad57ee12c467a97ca41e329283e3d2c50661097baa24256f\": rpc error: code = NotFound desc = could not find container \"2ad33abda3edcc56ad57ee12c467a97ca41e329283e3d2c50661097baa24256f\": container with ID starting with 2ad33abda3edcc56ad57ee12c467a97ca41e329283e3d2c50661097baa24256f not found: ID does not exist"
Aug 01 00:10:15.014934 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:15.014910    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2mj8r"]
Aug 01 00:10:15.186178 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:15.186145    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d6082ce-ee02-467d-8795-57f5f0b7ef06" path="/var/lib/kubelet/pods/8d6082ce-ee02-467d-8795-57f5f0b7ef06/volumes"
Aug 01 00:10:16.489506 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:16.489464    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd" podUID="18061ebd-edcd-436f-bb27-5aa163e88ce9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:10:26.489435 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:26.489388    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd" podUID="18061ebd-edcd-436f-bb27-5aa163e88ce9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:10:28.810222 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:28.810162    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd" podUID="18061ebd-edcd-436f-bb27-5aa163e88ce9" containerName="main" containerID="cri-o://9a4ad676478bd6a1337168747304052acb4af3c3baa1c0c7274f203c2d590a74" gracePeriod=45
Aug 01 00:10:29.068329 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:29.068256    2578 generic.go:356] "Generic (PLEG): container finished" podID="18061ebd-edcd-436f-bb27-5aa163e88ce9" containerID="9a4ad676478bd6a1337168747304052acb4af3c3baa1c0c7274f203c2d590a74" exitCode=0
Aug 01 00:10:29.068459 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:29.068329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd" event={"ID":"18061ebd-edcd-436f-bb27-5aa163e88ce9","Type":"ContainerDied","Data":"9a4ad676478bd6a1337168747304052acb4af3c3baa1c0c7274f203c2d590a74"}
Aug 01 00:10:29.068459 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:29.068408    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd" event={"ID":"18061ebd-edcd-436f-bb27-5aa163e88ce9","Type":"ContainerDied","Data":"018f7b118d0d1de908225372e50d8e76b27a0855cb58bde70f5ee8a7d8115955"}
Aug 01 00:10:29.068459 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:29.068425    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="018f7b118d0d1de908225372e50d8e76b27a0855cb58bde70f5ee8a7d8115955"
Aug 01 00:10:29.073654 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:29.073631    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd"
Aug 01 00:10:29.196162 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:29.196135    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfmtr\" (UniqueName: \"kubernetes.io/projected/18061ebd-edcd-436f-bb27-5aa163e88ce9-kube-api-access-gfmtr\") pod \"18061ebd-edcd-436f-bb27-5aa163e88ce9\" (UID: \"18061ebd-edcd-436f-bb27-5aa163e88ce9\") "
Aug 01 00:10:29.196322 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:29.196193    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18061ebd-edcd-436f-bb27-5aa163e88ce9-tls-certs\") pod \"18061ebd-edcd-436f-bb27-5aa163e88ce9\" (UID: \"18061ebd-edcd-436f-bb27-5aa163e88ce9\") "
Aug 01 00:10:29.198398 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:29.198375    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18061ebd-edcd-436f-bb27-5aa163e88ce9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18061ebd-edcd-436f-bb27-5aa163e88ce9" (UID: "18061ebd-edcd-436f-bb27-5aa163e88ce9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:10:29.198485 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:29.198429    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18061ebd-edcd-436f-bb27-5aa163e88ce9-kube-api-access-gfmtr" (OuterVolumeSpecName: "kube-api-access-gfmtr") pod "18061ebd-edcd-436f-bb27-5aa163e88ce9" (UID: "18061ebd-edcd-436f-bb27-5aa163e88ce9"). InnerVolumeSpecName "kube-api-access-gfmtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:10:29.296783 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:29.296709    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfmtr\" (UniqueName: \"kubernetes.io/projected/18061ebd-edcd-436f-bb27-5aa163e88ce9-kube-api-access-gfmtr\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:10:29.296783 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:29.296772    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18061ebd-edcd-436f-bb27-5aa163e88ce9-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:10:30.073964 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:30.073932    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd"
Aug 01 00:10:30.093512 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:30.093486    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd"]
Aug 01 00:10:30.096669 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:30.096642    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65p8jzd"]
Aug 01 00:10:31.185900 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:10:31.185864    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18061ebd-edcd-436f-bb27-5aa163e88ce9" path="/var/lib/kubelet/pods/18061ebd-edcd-436f-bb27-5aa163e88ce9/volumes"
Aug 01 00:11:01.256801 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:01.256768    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:11:01.257574 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:01.257544    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:11:45.244137 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.244101    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"]
Aug 01 00:11:45.244741 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.244707    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18061ebd-edcd-436f-bb27-5aa163e88ce9" containerName="main"
Aug 01 00:11:45.244794 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.244743    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="18061ebd-edcd-436f-bb27-5aa163e88ce9" containerName="main"
Aug 01 00:11:45.244794 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.244759    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d6082ce-ee02-467d-8795-57f5f0b7ef06" containerName="main"
Aug 01 00:11:45.244794 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.244768    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d6082ce-ee02-467d-8795-57f5f0b7ef06" containerName="main"
Aug 01 00:11:45.244794 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.244781    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d6082ce-ee02-467d-8795-57f5f0b7ef06" containerName="storage-initializer"
Aug 01 00:11:45.244794 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.244789    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d6082ce-ee02-467d-8795-57f5f0b7ef06" containerName="storage-initializer"
Aug 01 00:11:45.244953 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.244905    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="18061ebd-edcd-436f-bb27-5aa163e88ce9" containerName="main"
Aug 01 00:11:45.244953 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.244922    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d6082ce-ee02-467d-8795-57f5f0b7ef06" containerName="main"
Aug 01 00:11:45.247909 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.247887    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.250205 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.250180    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-z9g4d\""
Aug 01 00:11:45.250342 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.250182    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Aug 01 00:11:45.250342 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.250189    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\""
Aug 01 00:11:45.250342 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.250189    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Aug 01 00:11:45.257556 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.257527    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"]
Aug 01 00:11:45.383007 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.382966    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvjmg\" (UniqueName: \"kubernetes.io/projected/0ba587ee-549f-4bae-a856-6d8b6421cab8-kube-api-access-cvjmg\") pod \"test-div-v2-kserve-5f6fdb6b77-kpvdq\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.383194 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.383014    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-home\") pod \"test-div-v2-kserve-5f6fdb6b77-kpvdq\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.383194 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.383111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-kpvdq\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.383194 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.383162    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-kpvdq\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.383194 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.383191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-kpvdq\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.383330 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.383212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ba587ee-549f-4bae-a856-6d8b6421cab8-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-kpvdq\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.484021 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.483987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ba587ee-549f-4bae-a856-6d8b6421cab8-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-kpvdq\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.484194 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.484030    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvjmg\" (UniqueName: \"kubernetes.io/projected/0ba587ee-549f-4bae-a856-6d8b6421cab8-kube-api-access-cvjmg\") pod \"test-div-v2-kserve-5f6fdb6b77-kpvdq\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.484194 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.484052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-home\") pod \"test-div-v2-kserve-5f6fdb6b77-kpvdq\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.484194 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.484105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-kpvdq\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.484194 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.484138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-kpvdq\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.484194 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.484176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-kpvdq\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.484566 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.484531    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-kpvdq\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.484742 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.484590    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-home\") pod \"test-div-v2-kserve-5f6fdb6b77-kpvdq\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.484742 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.484600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-kpvdq\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.486514 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.486495    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-kpvdq\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.486613 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.486598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ba587ee-549f-4bae-a856-6d8b6421cab8-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-kpvdq\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.491150 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.491125    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvjmg\" (UniqueName: \"kubernetes.io/projected/0ba587ee-549f-4bae-a856-6d8b6421cab8-kube-api-access-cvjmg\") pod \"test-div-v2-kserve-5f6fdb6b77-kpvdq\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.562149 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.562061    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:45.701233 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.701203    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"]
Aug 01 00:11:45.704493 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:11:45.704463    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba587ee_549f_4bae_a856_6d8b6421cab8.slice/crio-4c1030747f4ce223ea4d7069f7268c3a9d6d6fe56cdee819d006bcbeb1f1c1b7 WatchSource:0}: Error finding container 4c1030747f4ce223ea4d7069f7268c3a9d6d6fe56cdee819d006bcbeb1f1c1b7: Status 404 returned error can't find the container with id 4c1030747f4ce223ea4d7069f7268c3a9d6d6fe56cdee819d006bcbeb1f1c1b7
Aug 01 00:11:45.706372 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:45.706356    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:11:46.485836 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:46.485797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq" event={"ID":"0ba587ee-549f-4bae-a856-6d8b6421cab8","Type":"ContainerStarted","Data":"4c1030747f4ce223ea4d7069f7268c3a9d6d6fe56cdee819d006bcbeb1f1c1b7"}
Aug 01 00:11:47.493649 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:47.493599    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq" event={"ID":"0ba587ee-549f-4bae-a856-6d8b6421cab8","Type":"ContainerStarted","Data":"ad20f2680aa7202b113022dd492de264ce5ed9df51263e73221d93791c8d3222"}
Aug 01 00:11:47.510638 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:47.510591    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq" podStartSLOduration=1.495430359 podStartE2EDuration="2.510575605s" totalImagesPullingTime="1.015145246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:11:47.508883207 +0000 UTC m=+1546.868774247" watchObservedRunningTime="2026-08-01 00:11:47.510575605 +0000 UTC m=+1546.870466674"
Aug 01 00:11:47.563164 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:47.563131    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:55.563115 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:55.563078    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:55.576026 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:55.575999    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:11:56.552968 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:11:56.552942    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:12:34.619295 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:34.619254    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"]
Aug 01 00:12:49.640243 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:49.640181    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq" podUID="0ba587ee-549f-4bae-a856-6d8b6421cab8" containerName="main" containerID="cri-o://ad20f2680aa7202b113022dd492de264ce5ed9df51263e73221d93791c8d3222" gracePeriod=45
Aug 01 00:12:49.838985 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:49.838949    2578 generic.go:356] "Generic (PLEG): container finished" podID="0ba587ee-549f-4bae-a856-6d8b6421cab8" containerID="ad20f2680aa7202b113022dd492de264ce5ed9df51263e73221d93791c8d3222" exitCode=0
Aug 01 00:12:49.839133 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:49.838998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq" event={"ID":"0ba587ee-549f-4bae-a856-6d8b6421cab8","Type":"ContainerDied","Data":"ad20f2680aa7202b113022dd492de264ce5ed9df51263e73221d93791c8d3222"}
Aug 01 00:12:49.895554 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:49.895529    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:12:49.950324 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:49.950290    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvjmg\" (UniqueName: \"kubernetes.io/projected/0ba587ee-549f-4bae-a856-6d8b6421cab8-kube-api-access-cvjmg\") pod \"0ba587ee-549f-4bae-a856-6d8b6421cab8\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") "
Aug 01 00:12:49.950511 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:49.950349    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-tmp-dir\") pod \"0ba587ee-549f-4bae-a856-6d8b6421cab8\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") "
Aug 01 00:12:49.950511 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:49.950413    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ba587ee-549f-4bae-a856-6d8b6421cab8-tls-certs\") pod \"0ba587ee-549f-4bae-a856-6d8b6421cab8\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") "
Aug 01 00:12:49.950511 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:49.950463    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-dshm\") pod \"0ba587ee-549f-4bae-a856-6d8b6421cab8\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") "
Aug 01 00:12:49.950511 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:49.950501    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-home\") pod \"0ba587ee-549f-4bae-a856-6d8b6421cab8\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") "
Aug 01 00:12:49.950770 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:49.950591    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0ba587ee-549f-4bae-a856-6d8b6421cab8" (UID: "0ba587ee-549f-4bae-a856-6d8b6421cab8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:12:49.950770 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:49.950624    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-model-cache\") pod \"0ba587ee-549f-4bae-a856-6d8b6421cab8\" (UID: \"0ba587ee-549f-4bae-a856-6d8b6421cab8\") "
Aug 01 00:12:49.950878 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:49.950778    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-home" (OuterVolumeSpecName: "home") pod "0ba587ee-549f-4bae-a856-6d8b6421cab8" (UID: "0ba587ee-549f-4bae-a856-6d8b6421cab8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:12:49.950939 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:49.950898    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ba587ee-549f-4bae-a856-6d8b6421cab8" (UID: "0ba587ee-549f-4bae-a856-6d8b6421cab8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:12:49.950998 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:49.950963    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-home\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:12:49.950998 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:49.950981    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-tmp-dir\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:12:49.952844 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:49.952809    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-dshm" (OuterVolumeSpecName: "dshm") pod "0ba587ee-549f-4bae-a856-6d8b6421cab8" (UID: "0ba587ee-549f-4bae-a856-6d8b6421cab8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:12:49.952974 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:49.952842    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba587ee-549f-4bae-a856-6d8b6421cab8-kube-api-access-cvjmg" (OuterVolumeSpecName: "kube-api-access-cvjmg") pod "0ba587ee-549f-4bae-a856-6d8b6421cab8" (UID: "0ba587ee-549f-4bae-a856-6d8b6421cab8"). InnerVolumeSpecName "kube-api-access-cvjmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:12:49.952974 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:49.952858    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ba587ee-549f-4bae-a856-6d8b6421cab8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ba587ee-549f-4bae-a856-6d8b6421cab8" (UID: "0ba587ee-549f-4bae-a856-6d8b6421cab8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:12:50.051686 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:50.051637    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ba587ee-549f-4bae-a856-6d8b6421cab8-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:12:50.051686 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:50.051667    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-dshm\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:12:50.051686 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:50.051685    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ba587ee-549f-4bae-a856-6d8b6421cab8-model-cache\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:12:50.051686 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:50.051695    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvjmg\" (UniqueName: \"kubernetes.io/projected/0ba587ee-549f-4bae-a856-6d8b6421cab8-kube-api-access-cvjmg\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:12:50.847145 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:50.847112    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"
Aug 01 00:12:50.847145 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:50.847128    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq" event={"ID":"0ba587ee-549f-4bae-a856-6d8b6421cab8","Type":"ContainerDied","Data":"4c1030747f4ce223ea4d7069f7268c3a9d6d6fe56cdee819d006bcbeb1f1c1b7"}
Aug 01 00:12:50.847733 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:50.847204    2578 scope.go:117] "RemoveContainer" containerID="ad20f2680aa7202b113022dd492de264ce5ed9df51263e73221d93791c8d3222"
Aug 01 00:12:50.866371 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:50.866349    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"]
Aug 01 00:12:50.871578 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:50.871558    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kpvdq"]
Aug 01 00:12:51.186523 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:12:51.186490    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ba587ee-549f-4bae-a856-6d8b6421cab8" path="/var/lib/kubelet/pods/0ba587ee-549f-4bae-a856-6d8b6421cab8/volumes"
Aug 01 00:13:00.140346 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.140262    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"]
Aug 01 00:13:00.140740 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.140648    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ba587ee-549f-4bae-a856-6d8b6421cab8" containerName="main"
Aug 01 00:13:00.140740 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.140659    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ba587ee-549f-4bae-a856-6d8b6421cab8" containerName="main"
Aug 01 00:13:00.140822 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.140745    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ba587ee-549f-4bae-a856-6d8b6421cab8" containerName="main"
Aug 01 00:13:00.145690 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.145669    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.153013 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.152969    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Aug 01 00:13:00.153178 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.153098    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Aug 01 00:13:00.153810 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.153331    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Aug 01 00:13:00.153810 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.153790    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-xdzmn\""
Aug 01 00:13:00.157629 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.157605    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"]
Aug 01 00:13:00.239322 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.239293    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-tnw87\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.239505 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.239341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-tnw87\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.239505 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.239413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f6eb52-cf5e-4146-b227-ec73eb36ec52-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-tnw87\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.239615 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.239496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvf9q\" (UniqueName: \"kubernetes.io/projected/73f6eb52-cf5e-4146-b227-ec73eb36ec52-kube-api-access-tvf9q\") pod \"test-div-v2-kserve-5f6fdb6b77-tnw87\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.239615 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.239533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-home\") pod \"test-div-v2-kserve-5f6fdb6b77-tnw87\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.239615 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.239561    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-tnw87\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.340096 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.340063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-home\") pod \"test-div-v2-kserve-5f6fdb6b77-tnw87\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.340285 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.340104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-tnw87\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.340285 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.340177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-tnw87\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.340285 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.340210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-tnw87\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.340285 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.340239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f6eb52-cf5e-4146-b227-ec73eb36ec52-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-tnw87\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.340285 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.340281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvf9q\" (UniqueName: \"kubernetes.io/projected/73f6eb52-cf5e-4146-b227-ec73eb36ec52-kube-api-access-tvf9q\") pod \"test-div-v2-kserve-5f6fdb6b77-tnw87\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.340547 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.340494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-home\") pod \"test-div-v2-kserve-5f6fdb6b77-tnw87\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.340607 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.340549    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-tnw87\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.340877 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.340852    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-tnw87\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.342843 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.342811    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-tnw87\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.343374 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.343350    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f6eb52-cf5e-4146-b227-ec73eb36ec52-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-tnw87\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.347445 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.347425    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvf9q\" (UniqueName: \"kubernetes.io/projected/73f6eb52-cf5e-4146-b227-ec73eb36ec52-kube-api-access-tvf9q\") pod \"test-div-v2-kserve-5f6fdb6b77-tnw87\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.462941 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.462903    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:00.602341 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.602315    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"]
Aug 01 00:13:00.603776 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:13:00.603750    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f6eb52_cf5e_4146_b227_ec73eb36ec52.slice/crio-b8f119730793ae8d1206f601849051be0b43aec64dfa4743ec2fea757745cbb2 WatchSource:0}: Error finding container b8f119730793ae8d1206f601849051be0b43aec64dfa4743ec2fea757745cbb2: Status 404 returned error can't find the container with id b8f119730793ae8d1206f601849051be0b43aec64dfa4743ec2fea757745cbb2
Aug 01 00:13:00.903988 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.903954    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87" event={"ID":"73f6eb52-cf5e-4146-b227-ec73eb36ec52","Type":"ContainerStarted","Data":"e9916e9fa8aab4af9eeaca41db2b4f56a4ca2de4bc698fd1c51d55437c14fd9c"}
Aug 01 00:13:00.904135 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.904009    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87" event={"ID":"73f6eb52-cf5e-4146-b227-ec73eb36ec52","Type":"ContainerStarted","Data":"b8f119730793ae8d1206f601849051be0b43aec64dfa4743ec2fea757745cbb2"}
Aug 01 00:13:00.922337 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:00.922277    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87" podStartSLOduration=0.922256074 podStartE2EDuration="922.256074ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:13:00.920961153 +0000 UTC m=+1620.280852231" watchObservedRunningTime="2026-08-01 00:13:00.922256074 +0000 UTC m=+1620.282147116"
Aug 01 00:13:01.463779 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:01.463736    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:10.463958 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:10.463917    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:10.477046 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:10.477017    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:10.972124 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:10.972085    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:29.775519 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:29.775486    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"]
Aug 01 00:13:44.797795 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:44.797733    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87" podUID="73f6eb52-cf5e-4146-b227-ec73eb36ec52" containerName="main" containerID="cri-o://e9916e9fa8aab4af9eeaca41db2b4f56a4ca2de4bc698fd1c51d55437c14fd9c" gracePeriod=45
Aug 01 00:13:44.961057 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:44.961012    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87" podUID="73f6eb52-cf5e-4146-b227-ec73eb36ec52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Aug 01 00:13:45.061009 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.060936    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:45.135265 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.134890    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-home\") pod \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") "
Aug 01 00:13:45.135265 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.134955    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-dshm\") pod \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") "
Aug 01 00:13:45.135265 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.134991    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-model-cache\") pod \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") "
Aug 01 00:13:45.135605 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.135329    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-model-cache" (OuterVolumeSpecName: "model-cache") pod "73f6eb52-cf5e-4146-b227-ec73eb36ec52" (UID: "73f6eb52-cf5e-4146-b227-ec73eb36ec52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:45.135605 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.135523    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-home" (OuterVolumeSpecName: "home") pod "73f6eb52-cf5e-4146-b227-ec73eb36ec52" (UID: "73f6eb52-cf5e-4146-b227-ec73eb36ec52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:45.137499 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.137477    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-dshm" (OuterVolumeSpecName: "dshm") pod "73f6eb52-cf5e-4146-b227-ec73eb36ec52" (UID: "73f6eb52-cf5e-4146-b227-ec73eb36ec52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:45.147133 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.147111    2578 generic.go:356] "Generic (PLEG): container finished" podID="73f6eb52-cf5e-4146-b227-ec73eb36ec52" containerID="e9916e9fa8aab4af9eeaca41db2b4f56a4ca2de4bc698fd1c51d55437c14fd9c" exitCode=0
Aug 01 00:13:45.147230 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.147158    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87" event={"ID":"73f6eb52-cf5e-4146-b227-ec73eb36ec52","Type":"ContainerDied","Data":"e9916e9fa8aab4af9eeaca41db2b4f56a4ca2de4bc698fd1c51d55437c14fd9c"}
Aug 01 00:13:45.147230 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.147174    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"
Aug 01 00:13:45.147230 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.147199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87" event={"ID":"73f6eb52-cf5e-4146-b227-ec73eb36ec52","Type":"ContainerDied","Data":"b8f119730793ae8d1206f601849051be0b43aec64dfa4743ec2fea757745cbb2"}
Aug 01 00:13:45.147230 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.147221    2578 scope.go:117] "RemoveContainer" containerID="e9916e9fa8aab4af9eeaca41db2b4f56a4ca2de4bc698fd1c51d55437c14fd9c"
Aug 01 00:13:45.156624 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.156609    2578 scope.go:117] "RemoveContainer" containerID="e9916e9fa8aab4af9eeaca41db2b4f56a4ca2de4bc698fd1c51d55437c14fd9c"
Aug 01 00:13:45.156875 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:13:45.156858    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9916e9fa8aab4af9eeaca41db2b4f56a4ca2de4bc698fd1c51d55437c14fd9c\": container with ID starting with e9916e9fa8aab4af9eeaca41db2b4f56a4ca2de4bc698fd1c51d55437c14fd9c not found: ID does not exist" containerID="e9916e9fa8aab4af9eeaca41db2b4f56a4ca2de4bc698fd1c51d55437c14fd9c"
Aug 01 00:13:45.156937 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.156883    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9916e9fa8aab4af9eeaca41db2b4f56a4ca2de4bc698fd1c51d55437c14fd9c"} err="failed to get container status \"e9916e9fa8aab4af9eeaca41db2b4f56a4ca2de4bc698fd1c51d55437c14fd9c\": rpc error: code = NotFound desc = could not find container \"e9916e9fa8aab4af9eeaca41db2b4f56a4ca2de4bc698fd1c51d55437c14fd9c\": container with ID starting with e9916e9fa8aab4af9eeaca41db2b4f56a4ca2de4bc698fd1c51d55437c14fd9c not found: ID does not exist"
Aug 01 00:13:45.236142 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.236104    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f6eb52-cf5e-4146-b227-ec73eb36ec52-tls-certs\") pod \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") "
Aug 01 00:13:45.236349 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.236165    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-tmp-dir\") pod \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") "
Aug 01 00:13:45.236349 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.236243    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvf9q\" (UniqueName: \"kubernetes.io/projected/73f6eb52-cf5e-4146-b227-ec73eb36ec52-kube-api-access-tvf9q\") pod \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\" (UID: \"73f6eb52-cf5e-4146-b227-ec73eb36ec52\") "
Aug 01 00:13:45.236472 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.236448    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "73f6eb52-cf5e-4146-b227-ec73eb36ec52" (UID: "73f6eb52-cf5e-4146-b227-ec73eb36ec52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:45.236529 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.236505    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-home\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:13:45.236529 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.236521    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-dshm\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:13:45.236633 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.236531    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-model-cache\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:13:45.236633 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.236546    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f6eb52-cf5e-4146-b227-ec73eb36ec52-tmp-dir\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:13:45.238462 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.238437    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f6eb52-cf5e-4146-b227-ec73eb36ec52-kube-api-access-tvf9q" (OuterVolumeSpecName: "kube-api-access-tvf9q") pod "73f6eb52-cf5e-4146-b227-ec73eb36ec52" (UID: "73f6eb52-cf5e-4146-b227-ec73eb36ec52"). InnerVolumeSpecName "kube-api-access-tvf9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:13:45.238462 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.238453    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73f6eb52-cf5e-4146-b227-ec73eb36ec52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "73f6eb52-cf5e-4146-b227-ec73eb36ec52" (UID: "73f6eb52-cf5e-4146-b227-ec73eb36ec52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:13:45.337362 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.337283    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvf9q\" (UniqueName: \"kubernetes.io/projected/73f6eb52-cf5e-4146-b227-ec73eb36ec52-kube-api-access-tvf9q\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:13:45.337362 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.337309    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f6eb52-cf5e-4146-b227-ec73eb36ec52-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:13:45.467690 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.467655    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"]
Aug 01 00:13:45.474036 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:45.474006    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-tnw87"]
Aug 01 00:13:47.185757 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:13:47.185698    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73f6eb52-cf5e-4146-b227-ec73eb36ec52" path="/var/lib/kubelet/pods/73f6eb52-cf5e-4146-b227-ec73eb36ec52/volumes"
Aug 01 00:15:01.384005 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:15:01.383975    2578 scope.go:117] "RemoveContainer" containerID="9a4ad676478bd6a1337168747304052acb4af3c3baa1c0c7274f203c2d590a74"
Aug 01 00:16:01.291762 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:01.291733    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:16:01.293947 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:01.293926    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:16:43.175880 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.175846    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59"]
Aug 01 00:16:43.176330 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.176260    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73f6eb52-cf5e-4146-b227-ec73eb36ec52" containerName="main"
Aug 01 00:16:43.176330 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.176279    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="73f6eb52-cf5e-4146-b227-ec73eb36ec52" containerName="main"
Aug 01 00:16:43.176403 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.176364    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="73f6eb52-cf5e-4146-b227-ec73eb36ec52" containerName="main"
Aug 01 00:16:43.179450 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.179431    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59"
Aug 01 00:16:43.181627 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.181597    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Aug 01 00:16:43.181804 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.181664    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Aug 01 00:16:43.181804 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.181699    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-t5l4c\""
Aug 01 00:16:43.181953 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.181812    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Aug 01 00:16:43.182017 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.181952    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-gcz9r\""
Aug 01 00:16:43.187219 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.187194    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59"]
Aug 01 00:16:43.229797 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.229744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skgs6\" (UniqueName: \"kubernetes.io/projected/a77273b2-010f-42b0-939b-0871540aab8c-kube-api-access-skgs6\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59\" (UID: \"a77273b2-010f-42b0-939b-0871540aab8c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59"
Aug 01 00:16:43.230075 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.229833    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77273b2-010f-42b0-939b-0871540aab8c-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59\" (UID: \"a77273b2-010f-42b0-939b-0871540aab8c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59"
Aug 01 00:16:43.330801 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.330765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skgs6\" (UniqueName: \"kubernetes.io/projected/a77273b2-010f-42b0-939b-0871540aab8c-kube-api-access-skgs6\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59\" (UID: \"a77273b2-010f-42b0-939b-0871540aab8c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59"
Aug 01 00:16:43.330961 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.330823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77273b2-010f-42b0-939b-0871540aab8c-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59\" (UID: \"a77273b2-010f-42b0-939b-0871540aab8c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59"
Aug 01 00:16:43.333586 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.333557    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77273b2-010f-42b0-939b-0871540aab8c-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59\" (UID: \"a77273b2-010f-42b0-939b-0871540aab8c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59"
Aug 01 00:16:43.337826 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.337800    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skgs6\" (UniqueName: \"kubernetes.io/projected/a77273b2-010f-42b0-939b-0871540aab8c-kube-api-access-skgs6\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59\" (UID: \"a77273b2-010f-42b0-939b-0871540aab8c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59"
Aug 01 00:16:43.493169 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.493076    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59"
Aug 01 00:16:43.638446 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:43.638421    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59"]
Aug 01 00:16:43.641270 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:16:43.641240    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77273b2_010f_42b0_939b_0871540aab8c.slice/crio-a85fbb9ef4f8158849b9b3ebebc6fdb7fe9e7469efa86c69faa4cb766e69bba0 WatchSource:0}: Error finding container a85fbb9ef4f8158849b9b3ebebc6fdb7fe9e7469efa86c69faa4cb766e69bba0: Status 404 returned error can't find the container with id a85fbb9ef4f8158849b9b3ebebc6fdb7fe9e7469efa86c69faa4cb766e69bba0
Aug 01 00:16:44.130614 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:44.130567    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59" event={"ID":"a77273b2-010f-42b0-939b-0871540aab8c","Type":"ContainerStarted","Data":"7ab9fcb6856ab5150c605fe407d4566d10f25e0b18cd7aa460d0bde0fdb535a0"}
Aug 01 00:16:44.130850 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:44.130625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59" event={"ID":"a77273b2-010f-42b0-939b-0871540aab8c","Type":"ContainerStarted","Data":"a85fbb9ef4f8158849b9b3ebebc6fdb7fe9e7469efa86c69faa4cb766e69bba0"}
Aug 01 00:16:44.130850 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:44.130787    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59"
Aug 01 00:16:44.146891 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:16:44.146839    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59" podStartSLOduration=1.146823974 podStartE2EDuration="1.146823974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:16:44.145649797 +0000 UTC m=+1843.505540839" watchObservedRunningTime="2026-08-01 00:16:44.146823974 +0000 UTC m=+1843.506715014"
Aug 01 00:17:15.138729 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:15.138692    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59"
Aug 01 00:17:35.672864 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:35.672829    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59"]
Aug 01 00:17:45.138545 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:45.138490    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59" podUID="a77273b2-010f-42b0-939b-0871540aab8c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:17:50.693918 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:50.693853    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59" podUID="a77273b2-010f-42b0-939b-0871540aab8c" containerName="main" containerID="cri-o://7ab9fcb6856ab5150c605fe407d4566d10f25e0b18cd7aa460d0bde0fdb535a0" gracePeriod=45
Aug 01 00:17:50.959669 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:50.959645    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59"
Aug 01 00:17:51.031059 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:51.031027    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77273b2-010f-42b0-939b-0871540aab8c-tls-certs\") pod \"a77273b2-010f-42b0-939b-0871540aab8c\" (UID: \"a77273b2-010f-42b0-939b-0871540aab8c\") "
Aug 01 00:17:51.031259 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:51.031070    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skgs6\" (UniqueName: \"kubernetes.io/projected/a77273b2-010f-42b0-939b-0871540aab8c-kube-api-access-skgs6\") pod \"a77273b2-010f-42b0-939b-0871540aab8c\" (UID: \"a77273b2-010f-42b0-939b-0871540aab8c\") "
Aug 01 00:17:51.033396 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:51.033365    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a77273b2-010f-42b0-939b-0871540aab8c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a77273b2-010f-42b0-939b-0871540aab8c" (UID: "a77273b2-010f-42b0-939b-0871540aab8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:17:51.033503 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:51.033397    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77273b2-010f-42b0-939b-0871540aab8c-kube-api-access-skgs6" (OuterVolumeSpecName: "kube-api-access-skgs6") pod "a77273b2-010f-42b0-939b-0871540aab8c" (UID: "a77273b2-010f-42b0-939b-0871540aab8c"). InnerVolumeSpecName "kube-api-access-skgs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:17:51.131861 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:51.131822    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77273b2-010f-42b0-939b-0871540aab8c-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:17:51.131861 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:51.131854    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skgs6\" (UniqueName: \"kubernetes.io/projected/a77273b2-010f-42b0-939b-0871540aab8c-kube-api-access-skgs6\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:17:51.505912 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:51.505878    2578 generic.go:356] "Generic (PLEG): container finished" podID="a77273b2-010f-42b0-939b-0871540aab8c" containerID="7ab9fcb6856ab5150c605fe407d4566d10f25e0b18cd7aa460d0bde0fdb535a0" exitCode=0
Aug 01 00:17:51.506101 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:51.505923    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59"
Aug 01 00:17:51.506101 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:51.505967    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59" event={"ID":"a77273b2-010f-42b0-939b-0871540aab8c","Type":"ContainerDied","Data":"7ab9fcb6856ab5150c605fe407d4566d10f25e0b18cd7aa460d0bde0fdb535a0"}
Aug 01 00:17:51.506101 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:51.506019    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59" event={"ID":"a77273b2-010f-42b0-939b-0871540aab8c","Type":"ContainerDied","Data":"a85fbb9ef4f8158849b9b3ebebc6fdb7fe9e7469efa86c69faa4cb766e69bba0"}
Aug 01 00:17:51.506101 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:51.506041    2578 scope.go:117] "RemoveContainer" containerID="7ab9fcb6856ab5150c605fe407d4566d10f25e0b18cd7aa460d0bde0fdb535a0"
Aug 01 00:17:51.515666 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:51.515643    2578 scope.go:117] "RemoveContainer" containerID="7ab9fcb6856ab5150c605fe407d4566d10f25e0b18cd7aa460d0bde0fdb535a0"
Aug 01 00:17:51.515934 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:17:51.515915    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab9fcb6856ab5150c605fe407d4566d10f25e0b18cd7aa460d0bde0fdb535a0\": container with ID starting with 7ab9fcb6856ab5150c605fe407d4566d10f25e0b18cd7aa460d0bde0fdb535a0 not found: ID does not exist" containerID="7ab9fcb6856ab5150c605fe407d4566d10f25e0b18cd7aa460d0bde0fdb535a0"
Aug 01 00:17:51.516016 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:51.515946    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab9fcb6856ab5150c605fe407d4566d10f25e0b18cd7aa460d0bde0fdb535a0"} err="failed to get container status \"7ab9fcb6856ab5150c605fe407d4566d10f25e0b18cd7aa460d0bde0fdb535a0\": rpc error: code = NotFound desc = could not find container \"7ab9fcb6856ab5150c605fe407d4566d10f25e0b18cd7aa460d0bde0fdb535a0\": container with ID starting with 7ab9fcb6856ab5150c605fe407d4566d10f25e0b18cd7aa460d0bde0fdb535a0 not found: ID does not exist"
Aug 01 00:17:51.520265 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:51.520244    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59"]
Aug 01 00:17:51.525382 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:51.525363    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepvf59"]
Aug 01 00:17:53.185455 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:17:53.185424    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a77273b2-010f-42b0-939b-0871540aab8c" path="/var/lib/kubelet/pods/a77273b2-010f-42b0-939b-0871540aab8c/volumes"
Aug 01 00:20:18.562239 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.562202    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"]
Aug 01 00:20:18.562763 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.562588    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a77273b2-010f-42b0-939b-0871540aab8c" containerName="main"
Aug 01 00:20:18.562763 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.562601    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a77273b2-010f-42b0-939b-0871540aab8c" containerName="main"
Aug 01 00:20:18.562763 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.562673    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a77273b2-010f-42b0-939b-0871540aab8c" containerName="main"
Aug 01 00:20:18.565845 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.565826    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.568023 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.567999    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Aug 01 00:20:18.568151 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.568002    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Aug 01 00:20:18.568151 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.568006    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-76kns\""
Aug 01 00:20:18.568151 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.568019    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Aug 01 00:20:18.574315 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.573603    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"]
Aug 01 00:20:18.645037 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.645000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.645229 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.645047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.645229 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.645110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.645229 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.645132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt76f\" (UniqueName: \"kubernetes.io/projected/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-kube-api-access-mt76f\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.645229 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.645161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.645411 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.645234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.746462 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.746418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.746639 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.746479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.746639 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.746505    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt76f\" (UniqueName: \"kubernetes.io/projected/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-kube-api-access-mt76f\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.746639 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.746534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.746639 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.746603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.746897 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.746660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.746960 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.746922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.746960 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.746939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.747069 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.747014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.749300 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.749253    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.750209 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.750181    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.759571 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.759544    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt76f\" (UniqueName: \"kubernetes.io/projected/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-kube-api-access-mt76f\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:18.879120 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:18.879035    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:19.036072 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:19.036042    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"]
Aug 01 00:20:19.037747 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:20:19.037700    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6dddd04_60dc_4e35_b05c_f2b3eea9022e.slice/crio-e621a1f2f743fefe8824db768bae1528e981cbdd41df3d37fa54606e7516179b WatchSource:0}: Error finding container e621a1f2f743fefe8824db768bae1528e981cbdd41df3d37fa54606e7516179b: Status 404 returned error can't find the container with id e621a1f2f743fefe8824db768bae1528e981cbdd41df3d37fa54606e7516179b
Aug 01 00:20:19.327071 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:19.327038    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m" event={"ID":"f6dddd04-60dc-4e35-b05c-f2b3eea9022e","Type":"ContainerStarted","Data":"62b97f7f948e476405967cb592b6220ca61615ddae5cc3571394fcad379e6783"}
Aug 01 00:20:19.327262 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:19.327092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m" event={"ID":"f6dddd04-60dc-4e35-b05c-f2b3eea9022e","Type":"ContainerStarted","Data":"e621a1f2f743fefe8824db768bae1528e981cbdd41df3d37fa54606e7516179b"}
Aug 01 00:20:19.345024 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:19.344978    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m" podStartSLOduration=1.344961871 podStartE2EDuration="1.344961871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:20:19.342895132 +0000 UTC m=+2058.702786174" watchObservedRunningTime="2026-08-01 00:20:19.344961871 +0000 UTC m=+2058.704852929"
Aug 01 00:20:19.879799 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:19.879759    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:28.880039 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:28.879944    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:28.892909 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:28.892879    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:20:29.392261 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:20:29.392227    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:21:01.327835 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:01.327803    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:21:01.329410 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:01.329388    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:21:30.310450 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:30.310412    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"]
Aug 01 00:21:30.322118 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:21:30.322077    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:21:30.322118 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:21:30.322118 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Aug 01 00:21:30.322118 ip-10-0-140-3 kubenswrapper[2578]:  > containerID="62b97f7f948e476405967cb592b6220ca61615ddae5cc3571394fcad379e6783" cmd=["/bin/sleep","15"]
Aug 01 00:21:30.322401 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:30.322137    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m" message="" err=<
Aug 01 00:21:30.322401 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:21:30.322401 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Aug 01 00:21:30.322401 ip-10-0-140-3 kubenswrapper[2578]:  >
Aug 01 00:21:30.322401 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:21:30.322150    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:21:30.322401 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:21:30.322401 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Aug 01 00:21:30.322401 ip-10-0-140-3 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m" podUID="f6dddd04-60dc-4e35-b05c-f2b3eea9022e" containerName="main" containerID="cri-o://62b97f7f948e476405967cb592b6220ca61615ddae5cc3571394fcad379e6783"
Aug 01 00:21:30.322401 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:30.322202    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m" podUID="f6dddd04-60dc-4e35-b05c-f2b3eea9022e" containerName="main" containerID="cri-o://62b97f7f948e476405967cb592b6220ca61615ddae5cc3571394fcad379e6783" gracePeriod=60
Aug 01 00:21:30.382041 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:30.382006    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m" podUID="f6dddd04-60dc-4e35-b05c-f2b3eea9022e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Aug 01 00:21:30.708760 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:30.708702    2578 generic.go:356] "Generic (PLEG): container finished" podID="f6dddd04-60dc-4e35-b05c-f2b3eea9022e" containerID="62b97f7f948e476405967cb592b6220ca61615ddae5cc3571394fcad379e6783" exitCode=0
Aug 01 00:21:30.708928 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:30.708787    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m" event={"ID":"f6dddd04-60dc-4e35-b05c-f2b3eea9022e","Type":"ContainerDied","Data":"62b97f7f948e476405967cb592b6220ca61615ddae5cc3571394fcad379e6783"}
Aug 01 00:21:31.068374 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.068345    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:21:31.165516 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.165483    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-model-cache\") pod \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") "
Aug 01 00:21:31.165708 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.165548    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-dshm\") pod \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") "
Aug 01 00:21:31.165708 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.165579    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-tmp-dir\") pod \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") "
Aug 01 00:21:31.165708 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.165596    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-tls-certs\") pod \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") "
Aug 01 00:21:31.165708 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.165617    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt76f\" (UniqueName: \"kubernetes.io/projected/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-kube-api-access-mt76f\") pod \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") "
Aug 01 00:21:31.165708 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.165661    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-home\") pod \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\" (UID: \"f6dddd04-60dc-4e35-b05c-f2b3eea9022e\") "
Aug 01 00:21:31.166022 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.165858    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6dddd04-60dc-4e35-b05c-f2b3eea9022e" (UID: "f6dddd04-60dc-4e35-b05c-f2b3eea9022e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:21:31.166022 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.165889    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6dddd04-60dc-4e35-b05c-f2b3eea9022e" (UID: "f6dddd04-60dc-4e35-b05c-f2b3eea9022e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:21:31.166022 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.165984    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-home" (OuterVolumeSpecName: "home") pod "f6dddd04-60dc-4e35-b05c-f2b3eea9022e" (UID: "f6dddd04-60dc-4e35-b05c-f2b3eea9022e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:21:31.166022 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.166005    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-model-cache\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:21:31.166022 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.166020    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-tmp-dir\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:21:31.168140 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.168100    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-kube-api-access-mt76f" (OuterVolumeSpecName: "kube-api-access-mt76f") pod "f6dddd04-60dc-4e35-b05c-f2b3eea9022e" (UID: "f6dddd04-60dc-4e35-b05c-f2b3eea9022e"). InnerVolumeSpecName "kube-api-access-mt76f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:21:31.168288 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.168150    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-dshm" (OuterVolumeSpecName: "dshm") pod "f6dddd04-60dc-4e35-b05c-f2b3eea9022e" (UID: "f6dddd04-60dc-4e35-b05c-f2b3eea9022e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:21:31.168288 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.168153    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6dddd04-60dc-4e35-b05c-f2b3eea9022e" (UID: "f6dddd04-60dc-4e35-b05c-f2b3eea9022e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:21:31.266810 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.266702    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt76f\" (UniqueName: \"kubernetes.io/projected/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-kube-api-access-mt76f\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:21:31.266810 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.266752    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-home\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:21:31.266810 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.266763    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-dshm\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:21:31.266810 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.266771    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6dddd04-60dc-4e35-b05c-f2b3eea9022e-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:21:31.716332 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.716303    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"
Aug 01 00:21:31.716879 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.716299    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m" event={"ID":"f6dddd04-60dc-4e35-b05c-f2b3eea9022e","Type":"ContainerDied","Data":"e621a1f2f743fefe8824db768bae1528e981cbdd41df3d37fa54606e7516179b"}
Aug 01 00:21:31.716879 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.716437    2578 scope.go:117] "RemoveContainer" containerID="62b97f7f948e476405967cb592b6220ca61615ddae5cc3571394fcad379e6783"
Aug 01 00:21:31.731945 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.731920    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"]
Aug 01 00:21:31.736399 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:31.736378    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cgg9m"]
Aug 01 00:21:33.185071 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:21:33.185036    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6dddd04-60dc-4e35-b05c-f2b3eea9022e" path="/var/lib/kubelet/pods/f6dddd04-60dc-4e35-b05c-f2b3eea9022e/volumes"
Aug 01 00:25:26.950898 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:26.950807    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"]
Aug 01 00:25:26.951398 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:26.951218    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6dddd04-60dc-4e35-b05c-f2b3eea9022e" containerName="main"
Aug 01 00:25:26.951398 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:26.951230    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6dddd04-60dc-4e35-b05c-f2b3eea9022e" containerName="main"
Aug 01 00:25:26.951398 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:26.951295    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6dddd04-60dc-4e35-b05c-f2b3eea9022e" containerName="main"
Aug 01 00:25:26.953960 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:26.953942    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:26.956036 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:26.956009    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Aug 01 00:25:26.956171 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:26.956008    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:25:26.956171 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:26.956009    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Aug 01 00:25:26.956171 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:26.956079    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-vvf2c\""
Aug 01 00:25:26.956336 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:26.956231    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Aug 01 00:25:26.956336 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:26.956287    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Aug 01 00:25:26.968302 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:26.968277    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"]
Aug 01 00:25:27.057101 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.057062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.057101 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.057105    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.057319 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.057159    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.057319 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.057200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk66t\" (UniqueName: \"kubernetes.io/projected/f868829e-13e2-449c-aaff-c26e76ae7c78-kube-api-access-mk66t\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.057319 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.057260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.057433 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.057320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f868829e-13e2-449c-aaff-c26e76ae7c78-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.057433 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.057355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f868829e-13e2-449c-aaff-c26e76ae7c78-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.057433 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.057374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.158590 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.158552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.158844 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.158613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f868829e-13e2-449c-aaff-c26e76ae7c78-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.158844 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.158645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f868829e-13e2-449c-aaff-c26e76ae7c78-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.158844 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.158671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.158844 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.158701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.158844 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.158764    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.158844 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.158805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.158844 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.158829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk66t\" (UniqueName: \"kubernetes.io/projected/f868829e-13e2-449c-aaff-c26e76ae7c78-kube-api-access-mk66t\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.159247 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.159052    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.159247 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.159129    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.159347 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.159251    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.159407 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.159374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f868829e-13e2-449c-aaff-c26e76ae7c78-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.159683 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.159641    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.161095 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.161077    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.161331 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.161314    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f868829e-13e2-449c-aaff-c26e76ae7c78-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.166830 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.166807    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk66t\" (UniqueName: \"kubernetes.io/projected/f868829e-13e2-449c-aaff-c26e76ae7c78-kube-api-access-mk66t\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.264482 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.264383    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:25:27.412917 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:27.412890    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"]
Aug 01 00:25:27.415084 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:25:27.415051    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf868829e_13e2_449c_aaff_c26e76ae7c78.slice/crio-2047a14f3c3da7d81df6f4fa6738bd3502eed05a0ee48f3db557a6878ab5717f WatchSource:0}: Error finding container 2047a14f3c3da7d81df6f4fa6738bd3502eed05a0ee48f3db557a6878ab5717f: Status 404 returned error can't find the container with id 2047a14f3c3da7d81df6f4fa6738bd3502eed05a0ee48f3db557a6878ab5717f
Aug 01 00:25:28.038149 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:28.038120    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl" event={"ID":"f868829e-13e2-449c-aaff-c26e76ae7c78","Type":"ContainerStarted","Data":"fb6185399ca7adeec5e8ea0d759b53aeda35612a7237126673a433cf79380581"}
Aug 01 00:25:28.038540 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:28.038169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl" event={"ID":"f868829e-13e2-449c-aaff-c26e76ae7c78","Type":"ContainerStarted","Data":"2047a14f3c3da7d81df6f4fa6738bd3502eed05a0ee48f3db557a6878ab5717f"}
Aug 01 00:25:31.057396 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:31.057363    2578 generic.go:356] "Generic (PLEG): container finished" podID="f868829e-13e2-449c-aaff-c26e76ae7c78" containerID="fb6185399ca7adeec5e8ea0d759b53aeda35612a7237126673a433cf79380581" exitCode=0
Aug 01 00:25:31.057396 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:31.057402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl" event={"ID":"f868829e-13e2-449c-aaff-c26e76ae7c78","Type":"ContainerDied","Data":"fb6185399ca7adeec5e8ea0d759b53aeda35612a7237126673a433cf79380581"}
Aug 01 00:25:31.058799 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:25:31.058783    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:26:01.284590 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:26:01.284555    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl" event={"ID":"f868829e-13e2-449c-aaff-c26e76ae7c78","Type":"ContainerStarted","Data":"0278885488de0fa5d88c53a0cffdbc6dc99802be294151e165fbe61991241a75"}
Aug 01 00:26:01.304747 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:26:01.304682    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl" podStartSLOduration=3.16926432 podStartE2EDuration="35.304664085s" totalImagesPullingTime="29.312583623s" totalInitContainerRuntime="2.822816142s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:26:01.302484315 +0000 UTC m=+2400.662375358" watchObservedRunningTime="2026-08-01 00:26:01.304664085 +0000 UTC m=+2400.664555125"
Aug 01 00:26:01.370791 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:26:01.370762    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:26:01.371311 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:26:01.371285    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:26:02.265152 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:26:02.265117    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:26:07.265242 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:26:07.265203    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:26:07.266694 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:26:07.266660    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl" podUID="f868829e-13e2-449c-aaff-c26e76ae7c78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Aug 01 00:26:17.265556 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:26:17.265511    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl" podUID="f868829e-13e2-449c-aaff-c26e76ae7c78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Aug 01 00:26:27.265551 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:26:27.265463    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl" podUID="f868829e-13e2-449c-aaff-c26e76ae7c78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Aug 01 00:26:37.265308 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:26:37.265267    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl" podUID="f868829e-13e2-449c-aaff-c26e76ae7c78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Aug 01 00:26:47.265098 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:26:47.265057    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl" podUID="f868829e-13e2-449c-aaff-c26e76ae7c78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Aug 01 00:26:57.265403 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:26:57.265365    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl" podUID="f868829e-13e2-449c-aaff-c26e76ae7c78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Aug 01 00:27:07.265794 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:27:07.265747    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl" podUID="f868829e-13e2-449c-aaff-c26e76ae7c78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Aug 01 00:27:17.274841 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:27:17.274801    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:27:17.282822 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:27:17.282797    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:27:50.866396 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:27:50.866352    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"]
Aug 01 00:28:05.890641 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:05.890534    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl" podUID="f868829e-13e2-449c-aaff-c26e76ae7c78" containerName="main" containerID="cri-o://0278885488de0fa5d88c53a0cffdbc6dc99802be294151e165fbe61991241a75" gracePeriod=45
Aug 01 00:28:06.275864 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.275814    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl" podUID="f868829e-13e2-449c-aaff-c26e76ae7c78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Aug 01 00:28:06.577653 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.577623    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:28:06.647615 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.647577    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-model-cache\") pod \"f868829e-13e2-449c-aaff-c26e76ae7c78\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") "
Aug 01 00:28:06.647841 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.647624    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-dshm\") pod \"f868829e-13e2-449c-aaff-c26e76ae7c78\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") "
Aug 01 00:28:06.647841 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.647655    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f868829e-13e2-449c-aaff-c26e76ae7c78-cabundle-cert\") pod \"f868829e-13e2-449c-aaff-c26e76ae7c78\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") "
Aug 01 00:28:06.647841 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.647684    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f868829e-13e2-449c-aaff-c26e76ae7c78-tls-certs\") pod \"f868829e-13e2-449c-aaff-c26e76ae7c78\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") "
Aug 01 00:28:06.647841 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.647768    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-kserve-provision-location\") pod \"f868829e-13e2-449c-aaff-c26e76ae7c78\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") "
Aug 01 00:28:06.647841 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.647839    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk66t\" (UniqueName: \"kubernetes.io/projected/f868829e-13e2-449c-aaff-c26e76ae7c78-kube-api-access-mk66t\") pod \"f868829e-13e2-449c-aaff-c26e76ae7c78\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") "
Aug 01 00:28:06.648114 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.647866    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-tmp-dir\") pod \"f868829e-13e2-449c-aaff-c26e76ae7c78\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") "
Aug 01 00:28:06.648114 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.647932    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-home\") pod \"f868829e-13e2-449c-aaff-c26e76ae7c78\" (UID: \"f868829e-13e2-449c-aaff-c26e76ae7c78\") "
Aug 01 00:28:06.648210 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.648079    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f868829e-13e2-449c-aaff-c26e76ae7c78-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f868829e-13e2-449c-aaff-c26e76ae7c78" (UID: "f868829e-13e2-449c-aaff-c26e76ae7c78"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:28:06.648291 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.648257    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f868829e-13e2-449c-aaff-c26e76ae7c78" (UID: "f868829e-13e2-449c-aaff-c26e76ae7c78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:28:06.648356 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.648306    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-model-cache" (OuterVolumeSpecName: "model-cache") pod "f868829e-13e2-449c-aaff-c26e76ae7c78" (UID: "f868829e-13e2-449c-aaff-c26e76ae7c78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:28:06.648590 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.648570    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-home" (OuterVolumeSpecName: "home") pod "f868829e-13e2-449c-aaff-c26e76ae7c78" (UID: "f868829e-13e2-449c-aaff-c26e76ae7c78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:28:06.650129 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.650102    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f868829e-13e2-449c-aaff-c26e76ae7c78-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f868829e-13e2-449c-aaff-c26e76ae7c78" (UID: "f868829e-13e2-449c-aaff-c26e76ae7c78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:28:06.650227 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.650186    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f868829e-13e2-449c-aaff-c26e76ae7c78-kube-api-access-mk66t" (OuterVolumeSpecName: "kube-api-access-mk66t") pod "f868829e-13e2-449c-aaff-c26e76ae7c78" (UID: "f868829e-13e2-449c-aaff-c26e76ae7c78"). InnerVolumeSpecName "kube-api-access-mk66t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:28:06.650291 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.650267    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-dshm" (OuterVolumeSpecName: "dshm") pod "f868829e-13e2-449c-aaff-c26e76ae7c78" (UID: "f868829e-13e2-449c-aaff-c26e76ae7c78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:28:06.669382 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.669349    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f868829e-13e2-449c-aaff-c26e76ae7c78" (UID: "f868829e-13e2-449c-aaff-c26e76ae7c78"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:28:06.749263 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.749225    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-home\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:28:06.749263 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.749256    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-model-cache\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:28:06.749263 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.749266    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-dshm\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:28:06.749263 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.749275    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f868829e-13e2-449c-aaff-c26e76ae7c78-cabundle-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:28:06.749533 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.749283    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f868829e-13e2-449c-aaff-c26e76ae7c78-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:28:06.749533 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.749291    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-kserve-provision-location\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:28:06.749533 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.749300    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk66t\" (UniqueName: \"kubernetes.io/projected/f868829e-13e2-449c-aaff-c26e76ae7c78-kube-api-access-mk66t\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:28:06.749533 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:06.749308    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f868829e-13e2-449c-aaff-c26e76ae7c78-tmp-dir\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:28:07.013113 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:07.013078    2578 generic.go:356] "Generic (PLEG): container finished" podID="f868829e-13e2-449c-aaff-c26e76ae7c78" containerID="0278885488de0fa5d88c53a0cffdbc6dc99802be294151e165fbe61991241a75" exitCode=0
Aug 01 00:28:07.013537 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:07.013143    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"
Aug 01 00:28:07.013537 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:07.013147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl" event={"ID":"f868829e-13e2-449c-aaff-c26e76ae7c78","Type":"ContainerDied","Data":"0278885488de0fa5d88c53a0cffdbc6dc99802be294151e165fbe61991241a75"}
Aug 01 00:28:07.013537 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:07.013212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl" event={"ID":"f868829e-13e2-449c-aaff-c26e76ae7c78","Type":"ContainerDied","Data":"2047a14f3c3da7d81df6f4fa6738bd3502eed05a0ee48f3db557a6878ab5717f"}
Aug 01 00:28:07.013537 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:07.013245    2578 scope.go:117] "RemoveContainer" containerID="0278885488de0fa5d88c53a0cffdbc6dc99802be294151e165fbe61991241a75"
Aug 01 00:28:07.023345 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:07.023327    2578 scope.go:117] "RemoveContainer" containerID="fb6185399ca7adeec5e8ea0d759b53aeda35612a7237126673a433cf79380581"
Aug 01 00:28:07.032368 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:07.032342    2578 scope.go:117] "RemoveContainer" containerID="0278885488de0fa5d88c53a0cffdbc6dc99802be294151e165fbe61991241a75"
Aug 01 00:28:07.032890 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:28:07.032744    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0278885488de0fa5d88c53a0cffdbc6dc99802be294151e165fbe61991241a75\": container with ID starting with 0278885488de0fa5d88c53a0cffdbc6dc99802be294151e165fbe61991241a75 not found: ID does not exist" containerID="0278885488de0fa5d88c53a0cffdbc6dc99802be294151e165fbe61991241a75"
Aug 01 00:28:07.032890 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:07.032786    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0278885488de0fa5d88c53a0cffdbc6dc99802be294151e165fbe61991241a75"} err="failed to get container status \"0278885488de0fa5d88c53a0cffdbc6dc99802be294151e165fbe61991241a75\": rpc error: code = NotFound desc = could not find container \"0278885488de0fa5d88c53a0cffdbc6dc99802be294151e165fbe61991241a75\": container with ID starting with 0278885488de0fa5d88c53a0cffdbc6dc99802be294151e165fbe61991241a75 not found: ID does not exist"
Aug 01 00:28:07.032890 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:07.032810    2578 scope.go:117] "RemoveContainer" containerID="fb6185399ca7adeec5e8ea0d759b53aeda35612a7237126673a433cf79380581"
Aug 01 00:28:07.033142 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:28:07.033107    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6185399ca7adeec5e8ea0d759b53aeda35612a7237126673a433cf79380581\": container with ID starting with fb6185399ca7adeec5e8ea0d759b53aeda35612a7237126673a433cf79380581 not found: ID does not exist" containerID="fb6185399ca7adeec5e8ea0d759b53aeda35612a7237126673a433cf79380581"
Aug 01 00:28:07.033241 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:07.033148    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6185399ca7adeec5e8ea0d759b53aeda35612a7237126673a433cf79380581"} err="failed to get container status \"fb6185399ca7adeec5e8ea0d759b53aeda35612a7237126673a433cf79380581\": rpc error: code = NotFound desc = could not find container \"fb6185399ca7adeec5e8ea0d759b53aeda35612a7237126673a433cf79380581\": container with ID starting with fb6185399ca7adeec5e8ea0d759b53aeda35612a7237126673a433cf79380581 not found: ID does not exist"
Aug 01 00:28:07.035500 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:07.035481    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"]
Aug 01 00:28:07.039360 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:07.039339    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5df4df58-nvjnl"]
Aug 01 00:28:07.185631 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:07.185600    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f868829e-13e2-449c-aaff-c26e76ae7c78" path="/var/lib/kubelet/pods/f868829e-13e2-449c-aaff-c26e76ae7c78/volumes"
Aug 01 00:28:44.859790 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.859752    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"]
Aug 01 00:28:44.860201 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.860142    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f868829e-13e2-449c-aaff-c26e76ae7c78" containerName="main"
Aug 01 00:28:44.860201 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.860154    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f868829e-13e2-449c-aaff-c26e76ae7c78" containerName="main"
Aug 01 00:28:44.860201 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.860163    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f868829e-13e2-449c-aaff-c26e76ae7c78" containerName="storage-initializer"
Aug 01 00:28:44.860201 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.860168    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f868829e-13e2-449c-aaff-c26e76ae7c78" containerName="storage-initializer"
Aug 01 00:28:44.860330 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.860241    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f868829e-13e2-449c-aaff-c26e76ae7c78" containerName="main"
Aug 01 00:28:44.863540 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.863517    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.865600 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.865577    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Aug 01 00:28:44.865732 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.865611    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Aug 01 00:28:44.865732 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.865632    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Aug 01 00:28:44.865732 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.865656    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Aug 01 00:28:44.865732 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.865661    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-7bstf\""
Aug 01 00:28:44.873845 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.873823    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"]
Aug 01 00:28:44.875148 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.875116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-dshm\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.875249 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.875150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.875249 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.875187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-home\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.875249 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.875209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-model-cache\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.875391 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.875313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.875391 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.875366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba82b4d6-1930-4ea7-9d57-1d736bf86389-tls-certs\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.875477 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.875391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9zh5\" (UniqueName: \"kubernetes.io/projected/ba82b4d6-1930-4ea7-9d57-1d736bf86389-kube-api-access-w9zh5\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.976363 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.976320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-home\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.976363 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.976363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-model-cache\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.976641 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.976417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.976641 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.976449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba82b4d6-1930-4ea7-9d57-1d736bf86389-tls-certs\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.976641 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.976466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9zh5\" (UniqueName: \"kubernetes.io/projected/ba82b4d6-1930-4ea7-9d57-1d736bf86389-kube-api-access-w9zh5\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.976641 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.976530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-dshm\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.976641 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.976557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.976955 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.976844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-home\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.976955 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.976899    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.976955 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.976939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-model-cache\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.977075 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.977007    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.978941 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.978917    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-dshm\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.979059 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.979043    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba82b4d6-1930-4ea7-9d57-1d736bf86389-tls-certs\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:44.983742 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:44.983696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9zh5\" (UniqueName: \"kubernetes.io/projected/ba82b4d6-1930-4ea7-9d57-1d736bf86389-kube-api-access-w9zh5\") pod \"e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:45.175461 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:45.175432    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:45.328090 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:45.328061    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"]
Aug 01 00:28:45.329699 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:28:45.329677    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba82b4d6_1930_4ea7_9d57_1d736bf86389.slice/crio-af699c24fce2fece77ffac563d7d71a358fb6783d6ee7fb82124575913387abb WatchSource:0}: Error finding container af699c24fce2fece77ffac563d7d71a358fb6783d6ee7fb82124575913387abb: Status 404 returned error can't find the container with id af699c24fce2fece77ffac563d7d71a358fb6783d6ee7fb82124575913387abb
Aug 01 00:28:46.230029 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:46.229985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6" event={"ID":"ba82b4d6-1930-4ea7-9d57-1d736bf86389","Type":"ContainerStarted","Data":"0c84a32dc0e5c813cf31e816909dfc89995ad5fe71e55638cf40253a3456a59e"}
Aug 01 00:28:46.230473 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:46.230091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6" event={"ID":"ba82b4d6-1930-4ea7-9d57-1d736bf86389","Type":"ContainerStarted","Data":"af699c24fce2fece77ffac563d7d71a358fb6783d6ee7fb82124575913387abb"}
Aug 01 00:28:48.265842 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:48.265797    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"]
Aug 01 00:28:48.266338 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:48.266064    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6" podUID="ba82b4d6-1930-4ea7-9d57-1d736bf86389" containerName="storage-initializer" containerID="cri-o://0c84a32dc0e5c813cf31e816909dfc89995ad5fe71e55638cf40253a3456a59e" gracePeriod=60
Aug 01 00:28:49.740665 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.740641    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:49.821626 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.821529    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-tmp-dir\") pod \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") "
Aug 01 00:28:49.821857 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.821629    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-home\") pod \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") "
Aug 01 00:28:49.821857 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.821765    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-model-cache\") pod \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") "
Aug 01 00:28:49.822009 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.821867    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-kserve-provision-location\") pod \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") "
Aug 01 00:28:49.822009 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.821900    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9zh5\" (UniqueName: \"kubernetes.io/projected/ba82b4d6-1930-4ea7-9d57-1d736bf86389-kube-api-access-w9zh5\") pod \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") "
Aug 01 00:28:49.822009 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.821866    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba82b4d6-1930-4ea7-9d57-1d736bf86389" (UID: "ba82b4d6-1930-4ea7-9d57-1d736bf86389"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:28:49.822009 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.821897    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-home" (OuterVolumeSpecName: "home") pod "ba82b4d6-1930-4ea7-9d57-1d736bf86389" (UID: "ba82b4d6-1930-4ea7-9d57-1d736bf86389"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:28:49.822009 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.821950    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba82b4d6-1930-4ea7-9d57-1d736bf86389-tls-certs\") pod \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") "
Aug 01 00:28:49.822009 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.821967    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-dshm\") pod \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\" (UID: \"ba82b4d6-1930-4ea7-9d57-1d736bf86389\") "
Aug 01 00:28:49.822314 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.822047    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba82b4d6-1930-4ea7-9d57-1d736bf86389" (UID: "ba82b4d6-1930-4ea7-9d57-1d736bf86389"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:28:49.822369 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.822350    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-tmp-dir\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:28:49.822421 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.822371    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-home\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:28:49.822421 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.822385    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-model-cache\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:28:49.824954 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.824900    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba82b4d6-1930-4ea7-9d57-1d736bf86389-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba82b4d6-1930-4ea7-9d57-1d736bf86389" (UID: "ba82b4d6-1930-4ea7-9d57-1d736bf86389"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:28:49.825203 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.825176    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-dshm" (OuterVolumeSpecName: "dshm") pod "ba82b4d6-1930-4ea7-9d57-1d736bf86389" (UID: "ba82b4d6-1930-4ea7-9d57-1d736bf86389"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:28:49.825338 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.825215    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba82b4d6-1930-4ea7-9d57-1d736bf86389-kube-api-access-w9zh5" (OuterVolumeSpecName: "kube-api-access-w9zh5") pod "ba82b4d6-1930-4ea7-9d57-1d736bf86389" (UID: "ba82b4d6-1930-4ea7-9d57-1d736bf86389"). InnerVolumeSpecName "kube-api-access-w9zh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:28:49.894582 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.894541    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ba82b4d6-1930-4ea7-9d57-1d736bf86389" (UID: "ba82b4d6-1930-4ea7-9d57-1d736bf86389"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:28:49.923511 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.923479    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-kserve-provision-location\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:28:49.923511 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.923507    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9zh5\" (UniqueName: \"kubernetes.io/projected/ba82b4d6-1930-4ea7-9d57-1d736bf86389-kube-api-access-w9zh5\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:28:49.923767 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.923520    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba82b4d6-1930-4ea7-9d57-1d736bf86389-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:28:49.923767 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:49.923528    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba82b4d6-1930-4ea7-9d57-1d736bf86389-dshm\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:28:50.257404 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:50.257364    2578 generic.go:356] "Generic (PLEG): container finished" podID="ba82b4d6-1930-4ea7-9d57-1d736bf86389" containerID="0c84a32dc0e5c813cf31e816909dfc89995ad5fe71e55638cf40253a3456a59e" exitCode=0
Aug 01 00:28:50.257617 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:50.257423    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"
Aug 01 00:28:50.257617 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:50.257451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6" event={"ID":"ba82b4d6-1930-4ea7-9d57-1d736bf86389","Type":"ContainerDied","Data":"0c84a32dc0e5c813cf31e816909dfc89995ad5fe71e55638cf40253a3456a59e"}
Aug 01 00:28:50.257617 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:50.257511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6" event={"ID":"ba82b4d6-1930-4ea7-9d57-1d736bf86389","Type":"ContainerDied","Data":"af699c24fce2fece77ffac563d7d71a358fb6783d6ee7fb82124575913387abb"}
Aug 01 00:28:50.257617 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:50.257533    2578 scope.go:117] "RemoveContainer" containerID="0c84a32dc0e5c813cf31e816909dfc89995ad5fe71e55638cf40253a3456a59e"
Aug 01 00:28:50.267547 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:50.267519    2578 scope.go:117] "RemoveContainer" containerID="0c84a32dc0e5c813cf31e816909dfc89995ad5fe71e55638cf40253a3456a59e"
Aug 01 00:28:50.267911 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:28:50.267881    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c84a32dc0e5c813cf31e816909dfc89995ad5fe71e55638cf40253a3456a59e\": container with ID starting with 0c84a32dc0e5c813cf31e816909dfc89995ad5fe71e55638cf40253a3456a59e not found: ID does not exist" containerID="0c84a32dc0e5c813cf31e816909dfc89995ad5fe71e55638cf40253a3456a59e"
Aug 01 00:28:50.268006 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:50.267919    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c84a32dc0e5c813cf31e816909dfc89995ad5fe71e55638cf40253a3456a59e"} err="failed to get container status \"0c84a32dc0e5c813cf31e816909dfc89995ad5fe71e55638cf40253a3456a59e\": rpc error: code = NotFound desc = could not find container \"0c84a32dc0e5c813cf31e816909dfc89995ad5fe71e55638cf40253a3456a59e\": container with ID starting with 0c84a32dc0e5c813cf31e816909dfc89995ad5fe71e55638cf40253a3456a59e not found: ID does not exist"
Aug 01 00:28:50.295420 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:50.295389    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"]
Aug 01 00:28:50.299001 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:50.298978    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-b5f4d6c7c-vgrz6"]
Aug 01 00:28:51.192748 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:28:51.192696    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba82b4d6-1930-4ea7-9d57-1d736bf86389" path="/var/lib/kubelet/pods/ba82b4d6-1930-4ea7-9d57-1d736bf86389/volumes"
Aug 01 00:29:00.964702 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:00.964668    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54"]
Aug 01 00:29:00.965207 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:00.965103    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba82b4d6-1930-4ea7-9d57-1d736bf86389" containerName="storage-initializer"
Aug 01 00:29:00.965207 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:00.965116    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba82b4d6-1930-4ea7-9d57-1d736bf86389" containerName="storage-initializer"
Aug 01 00:29:00.965207 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:00.965192    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba82b4d6-1930-4ea7-9d57-1d736bf86389" containerName="storage-initializer"
Aug 01 00:29:00.969989 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:00.969967    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54"
Aug 01 00:29:00.971653 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:00.971627    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"seaweedfs-s3-creds\""
Aug 01 00:29:00.971814 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:00.971651    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Aug 01 00:29:00.971814 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:00.971708    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-qg6cx\""
Aug 01 00:29:00.972071 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:00.972055    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Aug 01 00:29:00.976632 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:00.976612    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54"]
Aug 01 00:29:01.019467 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:01.019437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wg45\" (UniqueName: \"kubernetes.io/projected/fce4ad3c-4387-4f43-84ff-1c9e78ffdd09-kube-api-access-2wg45\") pod \"e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54\" (UID: \"fce4ad3c-4387-4f43-84ff-1c9e78ffdd09\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54"
Aug 01 00:29:01.019467 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:01.019475    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce4ad3c-4387-4f43-84ff-1c9e78ffdd09-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54\" (UID: \"fce4ad3c-4387-4f43-84ff-1c9e78ffdd09\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54"
Aug 01 00:29:01.120671 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:01.120634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wg45\" (UniqueName: \"kubernetes.io/projected/fce4ad3c-4387-4f43-84ff-1c9e78ffdd09-kube-api-access-2wg45\") pod \"e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54\" (UID: \"fce4ad3c-4387-4f43-84ff-1c9e78ffdd09\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54"
Aug 01 00:29:01.120671 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:01.120672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce4ad3c-4387-4f43-84ff-1c9e78ffdd09-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54\" (UID: \"fce4ad3c-4387-4f43-84ff-1c9e78ffdd09\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54"
Aug 01 00:29:01.121016 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:01.120997    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce4ad3c-4387-4f43-84ff-1c9e78ffdd09-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54\" (UID: \"fce4ad3c-4387-4f43-84ff-1c9e78ffdd09\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54"
Aug 01 00:29:01.127167 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:01.127149    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Aug 01 00:29:01.137407 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:01.137387    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Aug 01 00:29:01.149204 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:01.149180    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wg45\" (UniqueName: \"kubernetes.io/projected/fce4ad3c-4387-4f43-84ff-1c9e78ffdd09-kube-api-access-2wg45\") pod \"e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54\" (UID: \"fce4ad3c-4387-4f43-84ff-1c9e78ffdd09\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54"
Aug 01 00:29:01.285083 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:01.284989    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-qg6cx\""
Aug 01 00:29:01.293370 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:01.293347    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54"
Aug 01 00:29:01.438658 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:01.438624    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54"]
Aug 01 00:29:01.441802 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:29:01.441769    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce4ad3c_4387_4f43_84ff_1c9e78ffdd09.slice/crio-77ef37e2879ccc56597632c0b442ff3956a1fa110879db2c7909c6a24ebf684c WatchSource:0}: Error finding container 77ef37e2879ccc56597632c0b442ff3956a1fa110879db2c7909c6a24ebf684c: Status 404 returned error can't find the container with id 77ef37e2879ccc56597632c0b442ff3956a1fa110879db2c7909c6a24ebf684c
Aug 01 00:29:01.446166 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:01.446146    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"seaweedfs-s3-creds\""
Aug 01 00:29:02.328730 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:02.328673    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54" event={"ID":"fce4ad3c-4387-4f43-84ff-1c9e78ffdd09","Type":"ContainerStarted","Data":"e29d88fd533f93a915e8a994ad8467237e35c9275ebbcad4f41acdf49610e27c"}
Aug 01 00:29:02.329201 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:02.328771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54" event={"ID":"fce4ad3c-4387-4f43-84ff-1c9e78ffdd09","Type":"ContainerStarted","Data":"77ef37e2879ccc56597632c0b442ff3956a1fa110879db2c7909c6a24ebf684c"}
Aug 01 00:29:04.068056 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:04.068019    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54"]
Aug 01 00:29:04.340728 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:04.340596    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54" podUID="fce4ad3c-4387-4f43-84ff-1c9e78ffdd09" containerName="storage-initializer" containerID="cri-o://e29d88fd533f93a915e8a994ad8467237e35c9275ebbcad4f41acdf49610e27c" gracePeriod=30
Aug 01 00:29:05.616328 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:05.616302    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54"
Aug 01 00:29:05.764447 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:05.764362    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce4ad3c-4387-4f43-84ff-1c9e78ffdd09-kserve-provision-location\") pod \"fce4ad3c-4387-4f43-84ff-1c9e78ffdd09\" (UID: \"fce4ad3c-4387-4f43-84ff-1c9e78ffdd09\") "
Aug 01 00:29:05.764447 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:05.764436    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wg45\" (UniqueName: \"kubernetes.io/projected/fce4ad3c-4387-4f43-84ff-1c9e78ffdd09-kube-api-access-2wg45\") pod \"fce4ad3c-4387-4f43-84ff-1c9e78ffdd09\" (UID: \"fce4ad3c-4387-4f43-84ff-1c9e78ffdd09\") "
Aug 01 00:29:05.772543 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:05.772506    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce4ad3c-4387-4f43-84ff-1c9e78ffdd09-kube-api-access-2wg45" (OuterVolumeSpecName: "kube-api-access-2wg45") pod "fce4ad3c-4387-4f43-84ff-1c9e78ffdd09" (UID: "fce4ad3c-4387-4f43-84ff-1c9e78ffdd09"). InnerVolumeSpecName "kube-api-access-2wg45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:29:05.841041 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:05.840993    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce4ad3c-4387-4f43-84ff-1c9e78ffdd09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fce4ad3c-4387-4f43-84ff-1c9e78ffdd09" (UID: "fce4ad3c-4387-4f43-84ff-1c9e78ffdd09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:29:05.865415 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:05.865372    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce4ad3c-4387-4f43-84ff-1c9e78ffdd09-kserve-provision-location\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:29:05.865415 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:05.865411    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wg45\" (UniqueName: \"kubernetes.io/projected/fce4ad3c-4387-4f43-84ff-1c9e78ffdd09-kube-api-access-2wg45\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:29:06.352332 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:06.352300    2578 generic.go:356] "Generic (PLEG): container finished" podID="fce4ad3c-4387-4f43-84ff-1c9e78ffdd09" containerID="e29d88fd533f93a915e8a994ad8467237e35c9275ebbcad4f41acdf49610e27c" exitCode=0
Aug 01 00:29:06.352523 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:06.352350    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54"
Aug 01 00:29:06.352523 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:06.352401    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54" event={"ID":"fce4ad3c-4387-4f43-84ff-1c9e78ffdd09","Type":"ContainerDied","Data":"e29d88fd533f93a915e8a994ad8467237e35c9275ebbcad4f41acdf49610e27c"}
Aug 01 00:29:06.352523 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:06.352451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54" event={"ID":"fce4ad3c-4387-4f43-84ff-1c9e78ffdd09","Type":"ContainerDied","Data":"77ef37e2879ccc56597632c0b442ff3956a1fa110879db2c7909c6a24ebf684c"}
Aug 01 00:29:06.352523 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:06.352474    2578 scope.go:117] "RemoveContainer" containerID="e29d88fd533f93a915e8a994ad8467237e35c9275ebbcad4f41acdf49610e27c"
Aug 01 00:29:06.362517 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:06.362502    2578 scope.go:117] "RemoveContainer" containerID="e29d88fd533f93a915e8a994ad8467237e35c9275ebbcad4f41acdf49610e27c"
Aug 01 00:29:06.362805 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:29:06.362784    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e29d88fd533f93a915e8a994ad8467237e35c9275ebbcad4f41acdf49610e27c\": container with ID starting with e29d88fd533f93a915e8a994ad8467237e35c9275ebbcad4f41acdf49610e27c not found: ID does not exist" containerID="e29d88fd533f93a915e8a994ad8467237e35c9275ebbcad4f41acdf49610e27c"
Aug 01 00:29:06.362878 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:06.362814    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e29d88fd533f93a915e8a994ad8467237e35c9275ebbcad4f41acdf49610e27c"} err="failed to get container status \"e29d88fd533f93a915e8a994ad8467237e35c9275ebbcad4f41acdf49610e27c\": rpc error: code = NotFound desc = could not find container \"e29d88fd533f93a915e8a994ad8467237e35c9275ebbcad4f41acdf49610e27c\": container with ID starting with e29d88fd533f93a915e8a994ad8467237e35c9275ebbcad4f41acdf49610e27c not found: ID does not exist"
Aug 01 00:29:06.384527 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:06.384494    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54"]
Aug 01 00:29:06.387602 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:06.387580    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6c795f6998-52t54"]
Aug 01 00:29:07.186012 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:07.185982    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fce4ad3c-4387-4f43-84ff-1c9e78ffdd09" path="/var/lib/kubelet/pods/fce4ad3c-4387-4f43-84ff-1c9e78ffdd09/volumes"
Aug 01 00:29:20.374733 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.374681    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"]
Aug 01 00:29:20.375453 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.375423    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce4ad3c-4387-4f43-84ff-1c9e78ffdd09" containerName="storage-initializer"
Aug 01 00:29:20.375453 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.375453    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce4ad3c-4387-4f43-84ff-1c9e78ffdd09" containerName="storage-initializer"
Aug 01 00:29:20.375627 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.375540    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fce4ad3c-4387-4f43-84ff-1c9e78ffdd09" containerName="storage-initializer"
Aug 01 00:29:20.378985 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.378969    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.380980 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.380912    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Aug 01 00:29:20.380980 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.380950    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-tkj85\""
Aug 01 00:29:20.380980 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.380965    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Aug 01 00:29:20.381221 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.380950    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:29:20.381221 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.380950    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Aug 01 00:29:20.381221 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.380950    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Aug 01 00:29:20.390194 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.390170    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"]
Aug 01 00:29:20.499201 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.499169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5001d2b-5627-42f6-a37d-d499720bbabc-cabundle-cert\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.499201 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.499205    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-model-cache\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.499420 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.499237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-dshm\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.499420 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.499329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5001d2b-5627-42f6-a37d-d499720bbabc-tls-certs\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.499420 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.499361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdnm6\" (UniqueName: \"kubernetes.io/projected/b5001d2b-5627-42f6-a37d-d499720bbabc-kube-api-access-rdnm6\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.499420 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.499386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-tmp-dir\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.499551 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.499446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-kserve-provision-location\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.499551 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.499471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-home\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.600521 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.600488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5001d2b-5627-42f6-a37d-d499720bbabc-tls-certs\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.600752 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.600527    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdnm6\" (UniqueName: \"kubernetes.io/projected/b5001d2b-5627-42f6-a37d-d499720bbabc-kube-api-access-rdnm6\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.600752 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.600560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-tmp-dir\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.600752 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.600621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-kserve-provision-location\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.600752 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.600647    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-home\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.600752 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.600698    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5001d2b-5627-42f6-a37d-d499720bbabc-cabundle-cert\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.600752 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.600735    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-model-cache\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.601119 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.600783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-dshm\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.601119 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.600988    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-tmp-dir\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.601242 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.601198    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-home\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.601289 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.601242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-model-cache\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.601492 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.601470    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-kserve-provision-location\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.601691 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.601667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5001d2b-5627-42f6-a37d-d499720bbabc-cabundle-cert\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.603696 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.603666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-dshm\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.604111 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.604089    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5001d2b-5627-42f6-a37d-d499720bbabc-tls-certs\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.610580 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.610554    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdnm6\" (UniqueName: \"kubernetes.io/projected/b5001d2b-5627-42f6-a37d-d499720bbabc-kube-api-access-rdnm6\") pod \"stop-feature-test-kserve-678bdcb4f5-hbmdp\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.690409 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.690367    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:20.837683 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:20.837632    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"]
Aug 01 00:29:20.839987 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:29:20.839958    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5001d2b_5627_42f6_a37d_d499720bbabc.slice/crio-37064b710ef373c6ab420d93a2c6b90c9c16d979b138a8e0788c8221d7c60af0 WatchSource:0}: Error finding container 37064b710ef373c6ab420d93a2c6b90c9c16d979b138a8e0788c8221d7c60af0: Status 404 returned error can't find the container with id 37064b710ef373c6ab420d93a2c6b90c9c16d979b138a8e0788c8221d7c60af0
Aug 01 00:29:21.447693 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:21.447648    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" event={"ID":"b5001d2b-5627-42f6-a37d-d499720bbabc","Type":"ContainerStarted","Data":"13c27ef9a9e2934aab89747b8a1187931010ec3651347da9fc6323191fecdd34"}
Aug 01 00:29:21.448236 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:21.447740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" event={"ID":"b5001d2b-5627-42f6-a37d-d499720bbabc","Type":"ContainerStarted","Data":"37064b710ef373c6ab420d93a2c6b90c9c16d979b138a8e0788c8221d7c60af0"}
Aug 01 00:29:23.459631 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:23.459543    2578 generic.go:356] "Generic (PLEG): container finished" podID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerID="13c27ef9a9e2934aab89747b8a1187931010ec3651347da9fc6323191fecdd34" exitCode=0
Aug 01 00:29:23.459631 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:23.459605    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" event={"ID":"b5001d2b-5627-42f6-a37d-d499720bbabc","Type":"ContainerDied","Data":"13c27ef9a9e2934aab89747b8a1187931010ec3651347da9fc6323191fecdd34"}
Aug 01 00:29:24.468583 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:24.468548    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" event={"ID":"b5001d2b-5627-42f6-a37d-d499720bbabc","Type":"ContainerStarted","Data":"cff2522e2e086debd2b8f3fdd5aca27adc155414ad1e70a929d0dddc3b075808"}
Aug 01 00:29:24.490359 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:24.490295    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" podStartSLOduration=2.196814647 podStartE2EDuration="4.490272055s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.293457408s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:24.488142816 +0000 UTC m=+2603.848033869" watchObservedRunningTime="2026-08-01 00:29:24.490272055 +0000 UTC m=+2603.850163096"
Aug 01 00:29:24.691292 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:24.691254    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:30.690792 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:30.690755    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:29:30.692573 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:30.692543    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Aug 01 00:29:40.691734 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:40.691677    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Aug 01 00:29:50.690937 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:29:50.690897    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Aug 01 00:30:00.691509 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:30:00.691465    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Aug 01 00:30:10.691268 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:30:10.691214    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Aug 01 00:30:20.691763 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:30:20.691703    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Aug 01 00:30:30.691531 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:30:30.691490    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Aug 01 00:30:40.691391 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:30:40.691339    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Aug 01 00:30:50.691023 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:30:50.690985    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Aug 01 00:31:00.690947 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:00.690855    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Aug 01 00:31:01.418492 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:01.418466    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:31:01.418887 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:01.418866    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:31:10.691425 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:10.691381    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Aug 01 00:31:20.691847 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:20.691799    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Aug 01 00:31:30.701459 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:30.701429    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:31:30.709189 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:30.709164    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:31:32.241854 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:32.241816    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"]
Aug 01 00:31:47.267804 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:47.267655    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="main" containerID="cri-o://cff2522e2e086debd2b8f3fdd5aca27adc155414ad1e70a929d0dddc3b075808" gracePeriod=45
Aug 01 00:31:47.702477 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:47.702428    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Aug 01 00:31:48.019354 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.019323    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:31:48.064280 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.064249    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-dshm\") pod \"b5001d2b-5627-42f6-a37d-d499720bbabc\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") "
Aug 01 00:31:48.064479 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.064319    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5001d2b-5627-42f6-a37d-d499720bbabc-tls-certs\") pod \"b5001d2b-5627-42f6-a37d-d499720bbabc\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") "
Aug 01 00:31:48.064479 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.064347    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-model-cache\") pod \"b5001d2b-5627-42f6-a37d-d499720bbabc\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") "
Aug 01 00:31:48.064479 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.064375    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-home\") pod \"b5001d2b-5627-42f6-a37d-d499720bbabc\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") "
Aug 01 00:31:48.064479 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.064430    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5001d2b-5627-42f6-a37d-d499720bbabc-cabundle-cert\") pod \"b5001d2b-5627-42f6-a37d-d499720bbabc\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") "
Aug 01 00:31:48.064479 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.064469    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdnm6\" (UniqueName: \"kubernetes.io/projected/b5001d2b-5627-42f6-a37d-d499720bbabc-kube-api-access-rdnm6\") pod \"b5001d2b-5627-42f6-a37d-d499720bbabc\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") "
Aug 01 00:31:48.064803 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.064492    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-kserve-provision-location\") pod \"b5001d2b-5627-42f6-a37d-d499720bbabc\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") "
Aug 01 00:31:48.064803 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.064552    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-tmp-dir\") pod \"b5001d2b-5627-42f6-a37d-d499720bbabc\" (UID: \"b5001d2b-5627-42f6-a37d-d499720bbabc\") "
Aug 01 00:31:48.064803 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.064696    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5001d2b-5627-42f6-a37d-d499720bbabc" (UID: "b5001d2b-5627-42f6-a37d-d499720bbabc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:31:48.064971 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.064847    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5001d2b-5627-42f6-a37d-d499720bbabc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b5001d2b-5627-42f6-a37d-d499720bbabc" (UID: "b5001d2b-5627-42f6-a37d-d499720bbabc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:31:48.064971 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.064891    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-model-cache\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:31:48.065435 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.065406    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-home" (OuterVolumeSpecName: "home") pod "b5001d2b-5627-42f6-a37d-d499720bbabc" (UID: "b5001d2b-5627-42f6-a37d-d499720bbabc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:31:48.066950 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.066927    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5001d2b-5627-42f6-a37d-d499720bbabc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5001d2b-5627-42f6-a37d-d499720bbabc" (UID: "b5001d2b-5627-42f6-a37d-d499720bbabc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:31:48.067108 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.067085    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-dshm" (OuterVolumeSpecName: "dshm") pod "b5001d2b-5627-42f6-a37d-d499720bbabc" (UID: "b5001d2b-5627-42f6-a37d-d499720bbabc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:31:48.067215 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.067192    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5001d2b-5627-42f6-a37d-d499720bbabc-kube-api-access-rdnm6" (OuterVolumeSpecName: "kube-api-access-rdnm6") pod "b5001d2b-5627-42f6-a37d-d499720bbabc" (UID: "b5001d2b-5627-42f6-a37d-d499720bbabc"). InnerVolumeSpecName "kube-api-access-rdnm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:31:48.077258 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.077234    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5001d2b-5627-42f6-a37d-d499720bbabc" (UID: "b5001d2b-5627-42f6-a37d-d499720bbabc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:31:48.087227 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.087200    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5001d2b-5627-42f6-a37d-d499720bbabc" (UID: "b5001d2b-5627-42f6-a37d-d499720bbabc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:31:48.166107 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.166072    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdnm6\" (UniqueName: \"kubernetes.io/projected/b5001d2b-5627-42f6-a37d-d499720bbabc-kube-api-access-rdnm6\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:31:48.166107 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.166099    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-kserve-provision-location\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:31:48.166324 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.166132    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-tmp-dir\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:31:48.166324 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.166142    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-dshm\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:31:48.166324 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.166151    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5001d2b-5627-42f6-a37d-d499720bbabc-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:31:48.166324 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.166159    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5001d2b-5627-42f6-a37d-d499720bbabc-home\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:31:48.166324 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.166167    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5001d2b-5627-42f6-a37d-d499720bbabc-cabundle-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:31:48.304925 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.304837    2578 generic.go:356] "Generic (PLEG): container finished" podID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerID="cff2522e2e086debd2b8f3fdd5aca27adc155414ad1e70a929d0dddc3b075808" exitCode=0
Aug 01 00:31:48.304925 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.304875    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" event={"ID":"b5001d2b-5627-42f6-a37d-d499720bbabc","Type":"ContainerDied","Data":"cff2522e2e086debd2b8f3fdd5aca27adc155414ad1e70a929d0dddc3b075808"}
Aug 01 00:31:48.304925 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.304903    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"
Aug 01 00:31:48.305460 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.304945    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp" event={"ID":"b5001d2b-5627-42f6-a37d-d499720bbabc","Type":"ContainerDied","Data":"37064b710ef373c6ab420d93a2c6b90c9c16d979b138a8e0788c8221d7c60af0"}
Aug 01 00:31:48.305460 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.304977    2578 scope.go:117] "RemoveContainer" containerID="cff2522e2e086debd2b8f3fdd5aca27adc155414ad1e70a929d0dddc3b075808"
Aug 01 00:31:48.317314 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.317295    2578 scope.go:117] "RemoveContainer" containerID="13c27ef9a9e2934aab89747b8a1187931010ec3651347da9fc6323191fecdd34"
Aug 01 00:31:48.326000 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.325979    2578 scope.go:117] "RemoveContainer" containerID="cff2522e2e086debd2b8f3fdd5aca27adc155414ad1e70a929d0dddc3b075808"
Aug 01 00:31:48.326387 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:31:48.326340    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff2522e2e086debd2b8f3fdd5aca27adc155414ad1e70a929d0dddc3b075808\": container with ID starting with cff2522e2e086debd2b8f3fdd5aca27adc155414ad1e70a929d0dddc3b075808 not found: ID does not exist" containerID="cff2522e2e086debd2b8f3fdd5aca27adc155414ad1e70a929d0dddc3b075808"
Aug 01 00:31:48.326548 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.326377    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff2522e2e086debd2b8f3fdd5aca27adc155414ad1e70a929d0dddc3b075808"} err="failed to get container status \"cff2522e2e086debd2b8f3fdd5aca27adc155414ad1e70a929d0dddc3b075808\": rpc error: code = NotFound desc = could not find container \"cff2522e2e086debd2b8f3fdd5aca27adc155414ad1e70a929d0dddc3b075808\": container with ID starting with cff2522e2e086debd2b8f3fdd5aca27adc155414ad1e70a929d0dddc3b075808 not found: ID does not exist"
Aug 01 00:31:48.326548 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.326402    2578 scope.go:117] "RemoveContainer" containerID="13c27ef9a9e2934aab89747b8a1187931010ec3651347da9fc6323191fecdd34"
Aug 01 00:31:48.326853 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:31:48.326698    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c27ef9a9e2934aab89747b8a1187931010ec3651347da9fc6323191fecdd34\": container with ID starting with 13c27ef9a9e2934aab89747b8a1187931010ec3651347da9fc6323191fecdd34 not found: ID does not exist" containerID="13c27ef9a9e2934aab89747b8a1187931010ec3651347da9fc6323191fecdd34"
Aug 01 00:31:48.326937 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.326867    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c27ef9a9e2934aab89747b8a1187931010ec3651347da9fc6323191fecdd34"} err="failed to get container status \"13c27ef9a9e2934aab89747b8a1187931010ec3651347da9fc6323191fecdd34\": rpc error: code = NotFound desc = could not find container \"13c27ef9a9e2934aab89747b8a1187931010ec3651347da9fc6323191fecdd34\": container with ID starting with 13c27ef9a9e2934aab89747b8a1187931010ec3651347da9fc6323191fecdd34 not found: ID does not exist"
Aug 01 00:31:48.329094 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.329073    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"]
Aug 01 00:31:48.333288 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.333266    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-hbmdp"]
Aug 01 00:31:48.675739 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.675676    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"]
Aug 01 00:31:48.676318 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.676294    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="main"
Aug 01 00:31:48.676433 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.676321    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="main"
Aug 01 00:31:48.676433 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.676355    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="storage-initializer"
Aug 01 00:31:48.676433 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.676367    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="storage-initializer"
Aug 01 00:31:48.676433 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.676427    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" containerName="main"
Aug 01 00:31:48.681312 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.681292    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.683106 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.683081    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Aug 01 00:31:48.683230 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.683132    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Aug 01 00:31:48.683230 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.683143    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Aug 01 00:31:48.683230 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.683192    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-tkj85\""
Aug 01 00:31:48.683428 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.683237    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Aug 01 00:31:48.683428 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.683266    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:31:48.688912 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.688887    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"]
Aug 01 00:31:48.771769 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.771708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv67s\" (UniqueName: \"kubernetes.io/projected/4af66992-92cf-4819-814a-9852401986aa-kube-api-access-fv67s\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.771943 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.771782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-tmp-dir\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.771943 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.771828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4af66992-92cf-4819-814a-9852401986aa-cabundle-cert\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.771943 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.771901    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-model-cache\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.771943 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.771939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-dshm\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.772110 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.771973    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af66992-92cf-4819-814a-9852401986aa-tls-certs\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.772110 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.771989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-home\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.772110 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.772009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-kserve-provision-location\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.873439 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.873402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-dshm\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.873603 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.873453    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af66992-92cf-4819-814a-9852401986aa-tls-certs\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.873603 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.873566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-home\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.873603 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.873597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-kserve-provision-location\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.873742 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.873646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv67s\" (UniqueName: \"kubernetes.io/projected/4af66992-92cf-4819-814a-9852401986aa-kube-api-access-fv67s\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.873742 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.873691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-tmp-dir\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.873856 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.873742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4af66992-92cf-4819-814a-9852401986aa-cabundle-cert\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.873856 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.873802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-model-cache\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.873975 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.873936    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-home\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.874037 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.873990    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-kserve-provision-location\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.874175 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.874157    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-tmp-dir\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.874320 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.874296    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-model-cache\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.874486 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.874471    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4af66992-92cf-4819-814a-9852401986aa-cabundle-cert\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.876019 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.875994    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-dshm\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.876123 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.876042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af66992-92cf-4819-814a-9852401986aa-tls-certs\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.880596 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.880577    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv67s\" (UniqueName: \"kubernetes.io/projected/4af66992-92cf-4819-814a-9852401986aa-kube-api-access-fv67s\") pod \"stop-feature-test-kserve-678bdcb4f5-vrbgl\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:48.993113 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:48.993026    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:49.139793 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:49.139764    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"]
Aug 01 00:31:49.142800 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:31:49.142773    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af66992_92cf_4819_814a_9852401986aa.slice/crio-48e0e1678eef29fe7f41c0eb912be311886fea125f2dd80ceff3d2275c15f905 WatchSource:0}: Error finding container 48e0e1678eef29fe7f41c0eb912be311886fea125f2dd80ceff3d2275c15f905: Status 404 returned error can't find the container with id 48e0e1678eef29fe7f41c0eb912be311886fea125f2dd80ceff3d2275c15f905
Aug 01 00:31:49.185845 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:49.185819    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5001d2b-5627-42f6-a37d-d499720bbabc" path="/var/lib/kubelet/pods/b5001d2b-5627-42f6-a37d-d499720bbabc/volumes"
Aug 01 00:31:49.313093 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:49.312998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" event={"ID":"4af66992-92cf-4819-814a-9852401986aa","Type":"ContainerStarted","Data":"fd0464db196d9f2cb9c62cea01074261304b1f76bd6083874a5db95fbbe05ec0"}
Aug 01 00:31:49.313093 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:49.313056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" event={"ID":"4af66992-92cf-4819-814a-9852401986aa","Type":"ContainerStarted","Data":"48e0e1678eef29fe7f41c0eb912be311886fea125f2dd80ceff3d2275c15f905"}
Aug 01 00:31:52.334503 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:52.334466    2578 generic.go:356] "Generic (PLEG): container finished" podID="4af66992-92cf-4819-814a-9852401986aa" containerID="fd0464db196d9f2cb9c62cea01074261304b1f76bd6083874a5db95fbbe05ec0" exitCode=0
Aug 01 00:31:52.335004 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:52.334541    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" event={"ID":"4af66992-92cf-4819-814a-9852401986aa","Type":"ContainerDied","Data":"fd0464db196d9f2cb9c62cea01074261304b1f76bd6083874a5db95fbbe05ec0"}
Aug 01 00:31:53.342406 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:53.342370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" event={"ID":"4af66992-92cf-4819-814a-9852401986aa","Type":"ContainerStarted","Data":"2e7c5586ce077f2e714c1a304dc5145e6e4c1f9c407fbe1dccac2baee730134b"}
Aug 01 00:31:53.363826 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:53.363774    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" podStartSLOduration=3.085790229 podStartE2EDuration="5.363757203s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.277966974s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:31:53.361065883 +0000 UTC m=+2752.720956924" watchObservedRunningTime="2026-08-01 00:31:53.363757203 +0000 UTC m=+2752.723648273"
Aug 01 00:31:53.993791 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:53.993756    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:58.993380 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:58.993336    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:31:58.994981 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:31:58.994955    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" podUID="4af66992-92cf-4819-814a-9852401986aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Aug 01 00:32:08.994400 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:32:08.994355    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" podUID="4af66992-92cf-4819-814a-9852401986aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Aug 01 00:32:18.994348 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:32:18.994295    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" podUID="4af66992-92cf-4819-814a-9852401986aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Aug 01 00:32:28.993879 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:32:28.993795    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" podUID="4af66992-92cf-4819-814a-9852401986aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Aug 01 00:32:38.994356 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:32:38.994316    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" podUID="4af66992-92cf-4819-814a-9852401986aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Aug 01 00:32:48.993628 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:32:48.993582    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" podUID="4af66992-92cf-4819-814a-9852401986aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Aug 01 00:32:58.993938 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:32:58.993894    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" podUID="4af66992-92cf-4819-814a-9852401986aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Aug 01 00:33:08.993839 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:33:08.993792    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" podUID="4af66992-92cf-4819-814a-9852401986aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Aug 01 00:33:18.994420 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:33:18.994374    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" podUID="4af66992-92cf-4819-814a-9852401986aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Aug 01 00:33:28.994086 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:33:28.994043    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" podUID="4af66992-92cf-4819-814a-9852401986aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Aug 01 00:33:38.994315 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:33:38.994266    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" podUID="4af66992-92cf-4819-814a-9852401986aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Aug 01 00:33:48.993732 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:33:48.993684    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" podUID="4af66992-92cf-4819-814a-9852401986aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Aug 01 00:33:59.004240 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:33:59.004154    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:33:59.012423 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:33:59.012393    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:34:00.137908 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:00.137874    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"]
Aug 01 00:34:15.161990 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:15.161931    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" podUID="4af66992-92cf-4819-814a-9852401986aa" containerName="main" containerID="cri-o://2e7c5586ce077f2e714c1a304dc5145e6e4c1f9c407fbe1dccac2baee730134b" gracePeriod=45
Aug 01 00:34:15.913360 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:15.913317    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:34:15.994985 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:15.994893    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-tmp-dir\") pod \"4af66992-92cf-4819-814a-9852401986aa\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") "
Aug 01 00:34:15.994985 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:15.994952    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-dshm\") pod \"4af66992-92cf-4819-814a-9852401986aa\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") "
Aug 01 00:34:15.994985 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:15.994978    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv67s\" (UniqueName: \"kubernetes.io/projected/4af66992-92cf-4819-814a-9852401986aa-kube-api-access-fv67s\") pod \"4af66992-92cf-4819-814a-9852401986aa\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") "
Aug 01 00:34:15.995300 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:15.995030    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4af66992-92cf-4819-814a-9852401986aa-cabundle-cert\") pod \"4af66992-92cf-4819-814a-9852401986aa\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") "
Aug 01 00:34:15.995300 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:15.995052    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af66992-92cf-4819-814a-9852401986aa-tls-certs\") pod \"4af66992-92cf-4819-814a-9852401986aa\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") "
Aug 01 00:34:15.995300 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:15.995080    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-model-cache\") pod \"4af66992-92cf-4819-814a-9852401986aa\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") "
Aug 01 00:34:15.995300 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:15.995110    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-home\") pod \"4af66992-92cf-4819-814a-9852401986aa\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") "
Aug 01 00:34:15.995300 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:15.995155    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-kserve-provision-location\") pod \"4af66992-92cf-4819-814a-9852401986aa\" (UID: \"4af66992-92cf-4819-814a-9852401986aa\") "
Aug 01 00:34:15.995574 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:15.995485    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af66992-92cf-4819-814a-9852401986aa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4af66992-92cf-4819-814a-9852401986aa" (UID: "4af66992-92cf-4819-814a-9852401986aa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:34:15.995649 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:15.995615    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "4af66992-92cf-4819-814a-9852401986aa" (UID: "4af66992-92cf-4819-814a-9852401986aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:34:15.996108 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:15.995974    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-home" (OuterVolumeSpecName: "home") pod "4af66992-92cf-4819-814a-9852401986aa" (UID: "4af66992-92cf-4819-814a-9852401986aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:34:15.997486 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:15.997456    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af66992-92cf-4819-814a-9852401986aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4af66992-92cf-4819-814a-9852401986aa" (UID: "4af66992-92cf-4819-814a-9852401986aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:34:15.997581 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:15.997504    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af66992-92cf-4819-814a-9852401986aa-kube-api-access-fv67s" (OuterVolumeSpecName: "kube-api-access-fv67s") pod "4af66992-92cf-4819-814a-9852401986aa" (UID: "4af66992-92cf-4819-814a-9852401986aa"). InnerVolumeSpecName "kube-api-access-fv67s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:34:15.997581 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:15.997561    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-dshm" (OuterVolumeSpecName: "dshm") pod "4af66992-92cf-4819-814a-9852401986aa" (UID: "4af66992-92cf-4819-814a-9852401986aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:34:16.007103 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.007080    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4af66992-92cf-4819-814a-9852401986aa" (UID: "4af66992-92cf-4819-814a-9852401986aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:34:16.020578 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.020540    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4af66992-92cf-4819-814a-9852401986aa" (UID: "4af66992-92cf-4819-814a-9852401986aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:34:16.096282 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.096249    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-home\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:34:16.096282 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.096280    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-kserve-provision-location\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:34:16.096496 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.096308    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-tmp-dir\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:34:16.096496 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.096318    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-dshm\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:34:16.096496 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.096326    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv67s\" (UniqueName: \"kubernetes.io/projected/4af66992-92cf-4819-814a-9852401986aa-kube-api-access-fv67s\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:34:16.096496 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.096336    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4af66992-92cf-4819-814a-9852401986aa-cabundle-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:34:16.096496 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.096344    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af66992-92cf-4819-814a-9852401986aa-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:34:16.096496 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.096351    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4af66992-92cf-4819-814a-9852401986aa-model-cache\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:34:16.173591 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.173557    2578 generic.go:356] "Generic (PLEG): container finished" podID="4af66992-92cf-4819-814a-9852401986aa" containerID="2e7c5586ce077f2e714c1a304dc5145e6e4c1f9c407fbe1dccac2baee730134b" exitCode=0
Aug 01 00:34:16.174028 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.173613    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"
Aug 01 00:34:16.174028 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.173641    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" event={"ID":"4af66992-92cf-4819-814a-9852401986aa","Type":"ContainerDied","Data":"2e7c5586ce077f2e714c1a304dc5145e6e4c1f9c407fbe1dccac2baee730134b"}
Aug 01 00:34:16.174028 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.173737    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl" event={"ID":"4af66992-92cf-4819-814a-9852401986aa","Type":"ContainerDied","Data":"48e0e1678eef29fe7f41c0eb912be311886fea125f2dd80ceff3d2275c15f905"}
Aug 01 00:34:16.174028 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.173764    2578 scope.go:117] "RemoveContainer" containerID="2e7c5586ce077f2e714c1a304dc5145e6e4c1f9c407fbe1dccac2baee730134b"
Aug 01 00:34:16.184451 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.184432    2578 scope.go:117] "RemoveContainer" containerID="fd0464db196d9f2cb9c62cea01074261304b1f76bd6083874a5db95fbbe05ec0"
Aug 01 00:34:16.193084 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.193032    2578 scope.go:117] "RemoveContainer" containerID="2e7c5586ce077f2e714c1a304dc5145e6e4c1f9c407fbe1dccac2baee730134b"
Aug 01 00:34:16.193399 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:34:16.193375    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7c5586ce077f2e714c1a304dc5145e6e4c1f9c407fbe1dccac2baee730134b\": container with ID starting with 2e7c5586ce077f2e714c1a304dc5145e6e4c1f9c407fbe1dccac2baee730134b not found: ID does not exist" containerID="2e7c5586ce077f2e714c1a304dc5145e6e4c1f9c407fbe1dccac2baee730134b"
Aug 01 00:34:16.193490 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.193408    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7c5586ce077f2e714c1a304dc5145e6e4c1f9c407fbe1dccac2baee730134b"} err="failed to get container status \"2e7c5586ce077f2e714c1a304dc5145e6e4c1f9c407fbe1dccac2baee730134b\": rpc error: code = NotFound desc = could not find container \"2e7c5586ce077f2e714c1a304dc5145e6e4c1f9c407fbe1dccac2baee730134b\": container with ID starting with 2e7c5586ce077f2e714c1a304dc5145e6e4c1f9c407fbe1dccac2baee730134b not found: ID does not exist"
Aug 01 00:34:16.193490 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.193424    2578 scope.go:117] "RemoveContainer" containerID="fd0464db196d9f2cb9c62cea01074261304b1f76bd6083874a5db95fbbe05ec0"
Aug 01 00:34:16.193690 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:34:16.193672    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0464db196d9f2cb9c62cea01074261304b1f76bd6083874a5db95fbbe05ec0\": container with ID starting with fd0464db196d9f2cb9c62cea01074261304b1f76bd6083874a5db95fbbe05ec0 not found: ID does not exist" containerID="fd0464db196d9f2cb9c62cea01074261304b1f76bd6083874a5db95fbbe05ec0"
Aug 01 00:34:16.193773 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.193695    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0464db196d9f2cb9c62cea01074261304b1f76bd6083874a5db95fbbe05ec0"} err="failed to get container status \"fd0464db196d9f2cb9c62cea01074261304b1f76bd6083874a5db95fbbe05ec0\": rpc error: code = NotFound desc = could not find container \"fd0464db196d9f2cb9c62cea01074261304b1f76bd6083874a5db95fbbe05ec0\": container with ID starting with fd0464db196d9f2cb9c62cea01074261304b1f76bd6083874a5db95fbbe05ec0 not found: ID does not exist"
Aug 01 00:34:16.194649 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.194625    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"]
Aug 01 00:34:16.198208 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:16.198186    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-678bdcb4f5-vrbgl"]
Aug 01 00:34:17.184847 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:17.184811    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4af66992-92cf-4819-814a-9852401986aa" path="/var/lib/kubelet/pods/4af66992-92cf-4819-814a-9852401986aa/volumes"
Aug 01 00:34:27.277227 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.277178    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"]
Aug 01 00:34:27.278415 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.278389    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af66992-92cf-4819-814a-9852401986aa" containerName="main"
Aug 01 00:34:27.278506 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.278419    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af66992-92cf-4819-814a-9852401986aa" containerName="main"
Aug 01 00:34:27.278506 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.278435    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af66992-92cf-4819-814a-9852401986aa" containerName="storage-initializer"
Aug 01 00:34:27.278506 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.278445    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af66992-92cf-4819-814a-9852401986aa" containerName="storage-initializer"
Aug 01 00:34:27.278610 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.278550    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4af66992-92cf-4819-814a-9852401986aa" containerName="main"
Aug 01 00:34:27.283878 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.283857    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.285740 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.285703    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Aug 01 00:34:27.285859 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.285736    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Aug 01 00:34:27.285859 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.285835    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Aug 01 00:34:27.285948 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.285872    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-8qw5v\""
Aug 01 00:34:27.286014 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.285992    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Aug 01 00:34:27.286074 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.286029    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:34:27.292576 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.292520    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"]
Aug 01 00:34:27.397346 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.397314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-home\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.397346 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.397352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-cabundle-cert\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.397595 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.397374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-dshm\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.397595 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.397470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-model-cache\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.397595 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.397513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfwsk\" (UniqueName: \"kubernetes.io/projected/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-kube-api-access-sfwsk\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.397595 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.397566    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-tmp-dir\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.397810 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.397609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.397810 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.397636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-tls-certs\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.498612 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.498581    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-tmp-dir\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.498839 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.498630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.498839 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.498757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-tls-certs\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.498839 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.498793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-home\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.498839 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.498818    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-cabundle-cert\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.499086 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.498993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-dshm\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.499086 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.499066    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-tmp-dir\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.499086 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.499081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-model-cache\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.499254 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.499086    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.499254 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.499140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfwsk\" (UniqueName: \"kubernetes.io/projected/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-kube-api-access-sfwsk\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.499254 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.499138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-home\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.499416 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.499301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-model-cache\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.499524 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.499506    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-cabundle-cert\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.501319 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.501299    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-dshm\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.501484 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.501468    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-tls-certs\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.507406 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.507381    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfwsk\" (UniqueName: \"kubernetes.io/projected/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-kube-api-access-sfwsk\") pod \"lora-single-adapter-test-kserve-7cf4974b47-4fx2q\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.597651 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.597553    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:27.750985 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:27.750953    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"]
Aug 01 00:34:27.752792 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:34:27.752763    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c12e06_c2fd_4464_a7a3_0b0837d45e6a.slice/crio-16e14ea10d664ba5fb96d1869e6a935728c2aba3f69ef47746eb5368b1f34928 WatchSource:0}: Error finding container 16e14ea10d664ba5fb96d1869e6a935728c2aba3f69ef47746eb5368b1f34928: Status 404 returned error can't find the container with id 16e14ea10d664ba5fb96d1869e6a935728c2aba3f69ef47746eb5368b1f34928
Aug 01 00:34:28.240849 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:28.240809    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q" event={"ID":"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a","Type":"ContainerStarted","Data":"e005becec0aefe2f9398d6044a916783347ebff1cecf4d23b72615c152e41c36"}
Aug 01 00:34:28.241038 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:28.240867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q" event={"ID":"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a","Type":"ContainerStarted","Data":"16e14ea10d664ba5fb96d1869e6a935728c2aba3f69ef47746eb5368b1f34928"}
Aug 01 00:34:31.257616 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:31.257583    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" containerID="e005becec0aefe2f9398d6044a916783347ebff1cecf4d23b72615c152e41c36" exitCode=0
Aug 01 00:34:31.258051 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:31.257641    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q" event={"ID":"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a","Type":"ContainerDied","Data":"e005becec0aefe2f9398d6044a916783347ebff1cecf4d23b72615c152e41c36"}
Aug 01 00:34:32.266348 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:32.266310    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q" event={"ID":"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a","Type":"ContainerStarted","Data":"7be56ce6d04e2996cbd53ae3fff2bc14136551c1cd11ab8da6367ecd3699c0bf"}
Aug 01 00:34:32.286858 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:32.286796    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q" podStartSLOduration=2.329317309 podStartE2EDuration="5.286776628s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.957459319s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:32.284812462 +0000 UTC m=+2911.644703518" watchObservedRunningTime="2026-08-01 00:34:32.286776628 +0000 UTC m=+2911.646667669"
Aug 01 00:34:32.597946 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:32.597653    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:37.598248 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:37.598204    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:34:37.599990 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:37.599947    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q" podUID="b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Aug 01 00:34:47.598454 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:47.598410    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q" podUID="b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Aug 01 00:34:57.598439 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:34:57.598381    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q" podUID="b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Aug 01 00:35:07.598601 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:35:07.598558    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q" podUID="b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Aug 01 00:35:17.598336 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:35:17.598288    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q" podUID="b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Aug 01 00:35:27.598200 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:35:27.598093    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q" podUID="b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Aug 01 00:35:37.598708 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:35:37.598665    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q" podUID="b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Aug 01 00:35:47.608546 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:35:47.608507    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:35:47.616567 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:35:47.616541    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:36:01.458765 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:01.458708    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:36:01.464911 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:01.464887    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:36:19.447514 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:19.447482    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-7cf4974b47-4fx2q_b2c12e06-c2fd-4464-a7a3-0b0837d45e6a/storage-initializer/0.log"
Aug 01 00:36:19.456154 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:19.456129    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-7cf4974b47-4fx2q_b2c12e06-c2fd-4464-a7a3-0b0837d45e6a/main/0.log"
Aug 01 00:36:20.360793 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:20.360759    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"]
Aug 01 00:36:35.383920 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:35.383856    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q" podUID="b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" containerName="main" containerID="cri-o://7be56ce6d04e2996cbd53ae3fff2bc14136551c1cd11ab8da6367ecd3699c0bf" gracePeriod=45
Aug 01 00:36:35.610127 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:35.610071    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q" podUID="b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Aug 01 00:36:36.002290 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.002256    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" containerID="7be56ce6d04e2996cbd53ae3fff2bc14136551c1cd11ab8da6367ecd3699c0bf" exitCode=0
Aug 01 00:36:36.002451 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.002329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q" event={"ID":"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a","Type":"ContainerDied","Data":"7be56ce6d04e2996cbd53ae3fff2bc14136551c1cd11ab8da6367ecd3699c0bf"}
Aug 01 00:36:36.052664 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.052640    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:36:36.195060 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.195025    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-cabundle-cert\") pod \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") "
Aug 01 00:36:36.195060 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.195075    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-home\") pod \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") "
Aug 01 00:36:36.195355 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.195100    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-tmp-dir\") pod \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") "
Aug 01 00:36:36.195355 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.195124    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfwsk\" (UniqueName: \"kubernetes.io/projected/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-kube-api-access-sfwsk\") pod \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") "
Aug 01 00:36:36.195355 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.195156    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-model-cache\") pod \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") "
Aug 01 00:36:36.195355 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.195178    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-tls-certs\") pod \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") "
Aug 01 00:36:36.195355 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.195236    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-dshm\") pod \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") "
Aug 01 00:36:36.195355 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.195283    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-kserve-provision-location\") pod \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\" (UID: \"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a\") "
Aug 01 00:36:36.195691 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.195472    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" (UID: "b2c12e06-c2fd-4464-a7a3-0b0837d45e6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:36:36.195691 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.195472    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" (UID: "b2c12e06-c2fd-4464-a7a3-0b0837d45e6a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:36:36.195691 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.195506    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-home" (OuterVolumeSpecName: "home") pod "b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" (UID: "b2c12e06-c2fd-4464-a7a3-0b0837d45e6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:36:36.195900 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.195884    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-cabundle-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:36:36.195965 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.195906    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-home\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:36:36.195965 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.195919    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-model-cache\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:36:36.197690 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.197657    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-dshm" (OuterVolumeSpecName: "dshm") pod "b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" (UID: "b2c12e06-c2fd-4464-a7a3-0b0837d45e6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:36:36.197840 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.197703    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-kube-api-access-sfwsk" (OuterVolumeSpecName: "kube-api-access-sfwsk") pod "b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" (UID: "b2c12e06-c2fd-4464-a7a3-0b0837d45e6a"). InnerVolumeSpecName "kube-api-access-sfwsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:36:36.197840 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.197748    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" (UID: "b2c12e06-c2fd-4464-a7a3-0b0837d45e6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:36:36.207677 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.207622    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" (UID: "b2c12e06-c2fd-4464-a7a3-0b0837d45e6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:36:36.221625 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.221598    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" (UID: "b2c12e06-c2fd-4464-a7a3-0b0837d45e6a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:36:36.297223 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.297188    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-dshm\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:36:36.297223 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.297219    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-kserve-provision-location\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:36:36.297223 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.297231    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-tmp-dir\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:36:36.297454 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.297241    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfwsk\" (UniqueName: \"kubernetes.io/projected/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-kube-api-access-sfwsk\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:36:36.297454 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:36.297249    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:36:37.009377 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:37.009344    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"
Aug 01 00:36:37.009846 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:37.009372    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q" event={"ID":"b2c12e06-c2fd-4464-a7a3-0b0837d45e6a","Type":"ContainerDied","Data":"16e14ea10d664ba5fb96d1869e6a935728c2aba3f69ef47746eb5368b1f34928"}
Aug 01 00:36:37.009846 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:37.009439    2578 scope.go:117] "RemoveContainer" containerID="7be56ce6d04e2996cbd53ae3fff2bc14136551c1cd11ab8da6367ecd3699c0bf"
Aug 01 00:36:37.019730 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:37.019688    2578 scope.go:117] "RemoveContainer" containerID="e005becec0aefe2f9398d6044a916783347ebff1cecf4d23b72615c152e41c36"
Aug 01 00:36:37.033211 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:37.033185    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"]
Aug 01 00:36:37.036952 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:37.036923    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7cf4974b47-4fx2q"]
Aug 01 00:36:37.185280 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:37.185248    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" path="/var/lib/kubelet/pods/b2c12e06-c2fd-4464-a7a3-0b0837d45e6a/volumes"
Aug 01 00:36:54.376056 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.375974    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"]
Aug 01 00:36:54.378385 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.376357    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" containerName="storage-initializer"
Aug 01 00:36:54.378385 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.376370    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" containerName="storage-initializer"
Aug 01 00:36:54.378385 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.376396    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" containerName="main"
Aug 01 00:36:54.378385 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.376401    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" containerName="main"
Aug 01 00:36:54.378385 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.376479    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2c12e06-c2fd-4464-a7a3-0b0837d45e6a" containerName="main"
Aug 01 00:36:54.379500 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.379484    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.381652 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.381441    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Aug 01 00:36:54.381652 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.381487    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-r2rdl\""
Aug 01 00:36:54.381652 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.381509    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Aug 01 00:36:54.381652 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.381519    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Aug 01 00:36:54.381652 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.381490    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Aug 01 00:36:54.381652 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.381444    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:36:54.389076 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.389053    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"]
Aug 01 00:36:54.560973 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.560933    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5ccb826-a4b9-4dec-a6e4-23958393ec86-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.560973 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.560977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-model-cache\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.561240 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.561069    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hgqx\" (UniqueName: \"kubernetes.io/projected/c5ccb826-a4b9-4dec-a6e4-23958393ec86-kube-api-access-7hgqx\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.561240 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.561100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.561240 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.561121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ccb826-a4b9-4dec-a6e4-23958393ec86-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.561240 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.561186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.561240 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.561229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-home\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.561508 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.561269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-dshm\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.661976 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.661943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hgqx\" (UniqueName: \"kubernetes.io/projected/c5ccb826-a4b9-4dec-a6e4-23958393ec86-kube-api-access-7hgqx\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.661976 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.661982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.662261 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.662002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ccb826-a4b9-4dec-a6e4-23958393ec86-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.662261 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.662026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.662261 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.662049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-home\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.662261 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.662095    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-dshm\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.662261 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.662141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5ccb826-a4b9-4dec-a6e4-23958393ec86-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.662261 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.662174    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-model-cache\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.662572 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.662435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.662572 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.662460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.662572 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.662554    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-model-cache\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.662762 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.662578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-home\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.663058 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.663033    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5ccb826-a4b9-4dec-a6e4-23958393ec86-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.664788 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.664768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-dshm\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.665348 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.665329    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ccb826-a4b9-4dec-a6e4-23958393ec86-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.668746 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.668694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hgqx\" (UniqueName: \"kubernetes.io/projected/c5ccb826-a4b9-4dec-a6e4-23958393ec86-kube-api-access-7hgqx\") pod \"lora-multiple-adapters-test-kserve-6fbf56844-rrgc4\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.690783 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.690745    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:36:54.837488 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:54.837459    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"]
Aug 01 00:36:54.839392 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:36:54.839363    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ccb826_a4b9_4dec_a6e4_23958393ec86.slice/crio-ec892c8769934d0e52bfa35cfd434f146017e067d82ec93eee9abaeb66b4cd90 WatchSource:0}: Error finding container ec892c8769934d0e52bfa35cfd434f146017e067d82ec93eee9abaeb66b4cd90: Status 404 returned error can't find the container with id ec892c8769934d0e52bfa35cfd434f146017e067d82ec93eee9abaeb66b4cd90
Aug 01 00:36:55.120044 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:55.119957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4" event={"ID":"c5ccb826-a4b9-4dec-a6e4-23958393ec86","Type":"ContainerStarted","Data":"a6cd1115f10dacbd7b766ca28582fea3c4c03b043a1f13d1cd23b323644768fa"}
Aug 01 00:36:55.120044 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:55.120010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4" event={"ID":"c5ccb826-a4b9-4dec-a6e4-23958393ec86","Type":"ContainerStarted","Data":"ec892c8769934d0e52bfa35cfd434f146017e067d82ec93eee9abaeb66b4cd90"}
Aug 01 00:36:58.141276 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:58.141244    2578 generic.go:356] "Generic (PLEG): container finished" podID="c5ccb826-a4b9-4dec-a6e4-23958393ec86" containerID="a6cd1115f10dacbd7b766ca28582fea3c4c03b043a1f13d1cd23b323644768fa" exitCode=0
Aug 01 00:36:58.141681 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:58.141319    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4" event={"ID":"c5ccb826-a4b9-4dec-a6e4-23958393ec86","Type":"ContainerDied","Data":"a6cd1115f10dacbd7b766ca28582fea3c4c03b043a1f13d1cd23b323644768fa"}
Aug 01 00:36:59.150600 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:59.150560    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4" event={"ID":"c5ccb826-a4b9-4dec-a6e4-23958393ec86","Type":"ContainerStarted","Data":"875e073f2d0c46f55b133813c2f74e9bbf383730e6b64ad5e2009c94e7328a07"}
Aug 01 00:36:59.171031 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:59.170979    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4" podStartSLOduration=2.000660717 podStartE2EDuration="5.170960521s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.170299804s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:36:59.167896994 +0000 UTC m=+3058.527788034" watchObservedRunningTime="2026-08-01 00:36:59.170960521 +0000 UTC m=+3058.530851562"
Aug 01 00:36:59.691776 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:36:59.691737    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:37:04.691586 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:37:04.691544    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:37:04.693302 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:37:04.693274    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4" podUID="c5ccb826-a4b9-4dec-a6e4-23958393ec86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Aug 01 00:37:14.691602 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:37:14.691522    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4" podUID="c5ccb826-a4b9-4dec-a6e4-23958393ec86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Aug 01 00:37:24.691290 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:37:24.691248    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4" podUID="c5ccb826-a4b9-4dec-a6e4-23958393ec86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Aug 01 00:37:34.691383 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:37:34.691342    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4" podUID="c5ccb826-a4b9-4dec-a6e4-23958393ec86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Aug 01 00:37:44.691575 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:37:44.691523    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4" podUID="c5ccb826-a4b9-4dec-a6e4-23958393ec86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Aug 01 00:37:54.691096 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:37:54.691049    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4" podUID="c5ccb826-a4b9-4dec-a6e4-23958393ec86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Aug 01 00:38:04.692166 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:38:04.692118    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4" podUID="c5ccb826-a4b9-4dec-a6e4-23958393ec86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Aug 01 00:38:14.701304 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:38:14.701271    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:38:14.709338 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:38:14.709308    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:38:53.563752 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:38:53.563699    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6fbf56844-rrgc4_c5ccb826-a4b9-4dec-a6e4-23958393ec86/storage-initializer/0.log"
Aug 01 00:38:53.581052 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:38:53.581025    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6fbf56844-rrgc4_c5ccb826-a4b9-4dec-a6e4-23958393ec86/main/0.log"
Aug 01 00:38:54.658390 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:38:54.658362    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"]
Aug 01 00:39:09.681429 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:09.681357    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4" podUID="c5ccb826-a4b9-4dec-a6e4-23958393ec86" containerName="main" containerID="cri-o://875e073f2d0c46f55b133813c2f74e9bbf383730e6b64ad5e2009c94e7328a07" gracePeriod=45
Aug 01 00:39:10.349930 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.349897    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:39:10.426782 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.426657    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-model-cache\") pod \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") "
Aug 01 00:39:10.426782 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.426700    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-kserve-provision-location\") pod \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") "
Aug 01 00:39:10.426782 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.426752    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hgqx\" (UniqueName: \"kubernetes.io/projected/c5ccb826-a4b9-4dec-a6e4-23958393ec86-kube-api-access-7hgqx\") pod \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") "
Aug 01 00:39:10.427077 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.426791    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-home\") pod \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") "
Aug 01 00:39:10.427077 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.426839    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ccb826-a4b9-4dec-a6e4-23958393ec86-tls-certs\") pod \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") "
Aug 01 00:39:10.427077 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.426870    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-tmp-dir\") pod \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") "
Aug 01 00:39:10.427077 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.426893    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5ccb826-a4b9-4dec-a6e4-23958393ec86-cabundle-cert\") pod \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") "
Aug 01 00:39:10.427077 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.426927    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-dshm\") pod \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\" (UID: \"c5ccb826-a4b9-4dec-a6e4-23958393ec86\") "
Aug 01 00:39:10.427077 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.427010    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-model-cache" (OuterVolumeSpecName: "model-cache") pod "c5ccb826-a4b9-4dec-a6e4-23958393ec86" (UID: "c5ccb826-a4b9-4dec-a6e4-23958393ec86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:39:10.427399 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.427136    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-home" (OuterVolumeSpecName: "home") pod "c5ccb826-a4b9-4dec-a6e4-23958393ec86" (UID: "c5ccb826-a4b9-4dec-a6e4-23958393ec86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:39:10.427399 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.427382    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-model-cache\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:39:10.427511 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.427405    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-home\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:39:10.427743 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.427484    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ccb826-a4b9-4dec-a6e4-23958393ec86-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c5ccb826-a4b9-4dec-a6e4-23958393ec86" (UID: "c5ccb826-a4b9-4dec-a6e4-23958393ec86"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:39:10.429312 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.429280    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ccb826-a4b9-4dec-a6e4-23958393ec86-kube-api-access-7hgqx" (OuterVolumeSpecName: "kube-api-access-7hgqx") pod "c5ccb826-a4b9-4dec-a6e4-23958393ec86" (UID: "c5ccb826-a4b9-4dec-a6e4-23958393ec86"). InnerVolumeSpecName "kube-api-access-7hgqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:39:10.429809 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.429372    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ccb826-a4b9-4dec-a6e4-23958393ec86-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c5ccb826-a4b9-4dec-a6e4-23958393ec86" (UID: "c5ccb826-a4b9-4dec-a6e4-23958393ec86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:39:10.429809 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.429429    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-dshm" (OuterVolumeSpecName: "dshm") pod "c5ccb826-a4b9-4dec-a6e4-23958393ec86" (UID: "c5ccb826-a4b9-4dec-a6e4-23958393ec86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:39:10.438948 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.438919    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c5ccb826-a4b9-4dec-a6e4-23958393ec86" (UID: "c5ccb826-a4b9-4dec-a6e4-23958393ec86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:39:10.453013 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.452979    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c5ccb826-a4b9-4dec-a6e4-23958393ec86" (UID: "c5ccb826-a4b9-4dec-a6e4-23958393ec86"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:39:10.528088 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.528049    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-tmp-dir\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:39:10.528088 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.528078    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5ccb826-a4b9-4dec-a6e4-23958393ec86-cabundle-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:39:10.528088 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.528089    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-dshm\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:39:10.528088 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.528098    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5ccb826-a4b9-4dec-a6e4-23958393ec86-kserve-provision-location\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:39:10.528405 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.528107    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hgqx\" (UniqueName: \"kubernetes.io/projected/c5ccb826-a4b9-4dec-a6e4-23958393ec86-kube-api-access-7hgqx\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:39:10.528405 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.528115    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ccb826-a4b9-4dec-a6e4-23958393ec86-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:39:10.946887 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.946851    2578 generic.go:356] "Generic (PLEG): container finished" podID="c5ccb826-a4b9-4dec-a6e4-23958393ec86" containerID="875e073f2d0c46f55b133813c2f74e9bbf383730e6b64ad5e2009c94e7328a07" exitCode=0
Aug 01 00:39:10.947424 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.946910    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"
Aug 01 00:39:10.947424 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.946938    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4" event={"ID":"c5ccb826-a4b9-4dec-a6e4-23958393ec86","Type":"ContainerDied","Data":"875e073f2d0c46f55b133813c2f74e9bbf383730e6b64ad5e2009c94e7328a07"}
Aug 01 00:39:10.947424 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.947038    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4" event={"ID":"c5ccb826-a4b9-4dec-a6e4-23958393ec86","Type":"ContainerDied","Data":"ec892c8769934d0e52bfa35cfd434f146017e067d82ec93eee9abaeb66b4cd90"}
Aug 01 00:39:10.947424 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.947074    2578 scope.go:117] "RemoveContainer" containerID="875e073f2d0c46f55b133813c2f74e9bbf383730e6b64ad5e2009c94e7328a07"
Aug 01 00:39:10.956992 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.956970    2578 scope.go:117] "RemoveContainer" containerID="a6cd1115f10dacbd7b766ca28582fea3c4c03b043a1f13d1cd23b323644768fa"
Aug 01 00:39:10.965542 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.965518    2578 scope.go:117] "RemoveContainer" containerID="875e073f2d0c46f55b133813c2f74e9bbf383730e6b64ad5e2009c94e7328a07"
Aug 01 00:39:10.966381 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:39:10.966358    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"875e073f2d0c46f55b133813c2f74e9bbf383730e6b64ad5e2009c94e7328a07\": container with ID starting with 875e073f2d0c46f55b133813c2f74e9bbf383730e6b64ad5e2009c94e7328a07 not found: ID does not exist" containerID="875e073f2d0c46f55b133813c2f74e9bbf383730e6b64ad5e2009c94e7328a07"
Aug 01 00:39:10.966503 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.966388    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875e073f2d0c46f55b133813c2f74e9bbf383730e6b64ad5e2009c94e7328a07"} err="failed to get container status \"875e073f2d0c46f55b133813c2f74e9bbf383730e6b64ad5e2009c94e7328a07\": rpc error: code = NotFound desc = could not find container \"875e073f2d0c46f55b133813c2f74e9bbf383730e6b64ad5e2009c94e7328a07\": container with ID starting with 875e073f2d0c46f55b133813c2f74e9bbf383730e6b64ad5e2009c94e7328a07 not found: ID does not exist"
Aug 01 00:39:10.966503 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.966407    2578 scope.go:117] "RemoveContainer" containerID="a6cd1115f10dacbd7b766ca28582fea3c4c03b043a1f13d1cd23b323644768fa"
Aug 01 00:39:10.966696 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:39:10.966678    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6cd1115f10dacbd7b766ca28582fea3c4c03b043a1f13d1cd23b323644768fa\": container with ID starting with a6cd1115f10dacbd7b766ca28582fea3c4c03b043a1f13d1cd23b323644768fa not found: ID does not exist" containerID="a6cd1115f10dacbd7b766ca28582fea3c4c03b043a1f13d1cd23b323644768fa"
Aug 01 00:39:10.966868 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.966703    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6cd1115f10dacbd7b766ca28582fea3c4c03b043a1f13d1cd23b323644768fa"} err="failed to get container status \"a6cd1115f10dacbd7b766ca28582fea3c4c03b043a1f13d1cd23b323644768fa\": rpc error: code = NotFound desc = could not find container \"a6cd1115f10dacbd7b766ca28582fea3c4c03b043a1f13d1cd23b323644768fa\": container with ID starting with a6cd1115f10dacbd7b766ca28582fea3c4c03b043a1f13d1cd23b323644768fa not found: ID does not exist"
Aug 01 00:39:10.968676 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.968616    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"]
Aug 01 00:39:10.971384 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:10.971365    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6fbf56844-rrgc4"]
Aug 01 00:39:11.185592 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:11.185554    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5ccb826-a4b9-4dec-a6e4-23958393ec86" path="/var/lib/kubelet/pods/c5ccb826-a4b9-4dec-a6e4-23958393ec86/volumes"
Aug 01 00:39:22.158165 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.158123    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"]
Aug 01 00:39:22.158771 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.158525    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5ccb826-a4b9-4dec-a6e4-23958393ec86" containerName="main"
Aug 01 00:39:22.158771 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.158539    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5ccb826-a4b9-4dec-a6e4-23958393ec86" containerName="main"
Aug 01 00:39:22.158771 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.158555    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5ccb826-a4b9-4dec-a6e4-23958393ec86" containerName="storage-initializer"
Aug 01 00:39:22.158771 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.158562    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5ccb826-a4b9-4dec-a6e4-23958393ec86" containerName="storage-initializer"
Aug 01 00:39:22.158771 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.158628    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5ccb826-a4b9-4dec-a6e4-23958393ec86" containerName="main"
Aug 01 00:39:22.162044 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.162024    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.163754 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.163709    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Aug 01 00:39:22.163903 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.163881    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Aug 01 00:39:22.163978 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.163923    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:39:22.164042 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.164025    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Aug 01 00:39:22.164102 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.164066    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-9gbdv\""
Aug 01 00:39:22.164160 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.164143    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Aug 01 00:39:22.171100 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.171081    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"]
Aug 01 00:39:22.232981 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.232944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-home\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.233206 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.233005    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm5ck\" (UniqueName: \"kubernetes.io/projected/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-kube-api-access-tm5ck\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.233301 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.233207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-model-cache\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.233301 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.233273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-tls-certs\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.233428 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.233298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-dshm\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.233428 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.233341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-cabundle-cert\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.233428 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.233402    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-tmp-dir\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.233568 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.233444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-kserve-provision-location\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.334258 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.334203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-home\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.334258 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.334262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm5ck\" (UniqueName: \"kubernetes.io/projected/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-kube-api-access-tm5ck\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.334534 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.334300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-model-cache\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.334534 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.334329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-tls-certs\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.334534 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.334348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-dshm\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.334534 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.334384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-cabundle-cert\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.334534 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.334421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-tmp-dir\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.334534 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.334447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-kserve-provision-location\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.334879 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.334695    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-home\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.334879 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.334760    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-model-cache\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.334879 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.334848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-kserve-provision-location\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.334986 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.334902    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-tmp-dir\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.335335 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.335313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-cabundle-cert\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.336847 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.336822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-dshm\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.337220 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.337197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-tls-certs\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.342043 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.342021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm5ck\" (UniqueName: \"kubernetes.io/projected/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-kube-api-access-tm5ck\") pod \"tls-verification-test-kserve-64cbb688cd-dwtr7\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.410982 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.410891    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw"]
Aug 01 00:39:22.415310 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.415287    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw"
Aug 01 00:39:22.417597 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.417571    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-k4l6n\""
Aug 01 00:39:22.424371 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.424339    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw"]
Aug 01 00:39:22.474304 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.474263    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:22.536821 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.536779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7661ef5b-e02a-4cc0-8680-9b65ede08a92-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-dsmqw\" (UID: \"7661ef5b-e02a-4cc0-8680-9b65ede08a92\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw"
Aug 01 00:39:22.536996 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.536855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clk6c\" (UniqueName: \"kubernetes.io/projected/7661ef5b-e02a-4cc0-8680-9b65ede08a92-kube-api-access-clk6c\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-dsmqw\" (UID: \"7661ef5b-e02a-4cc0-8680-9b65ede08a92\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw"
Aug 01 00:39:22.624885 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.624856    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"]
Aug 01 00:39:22.637383 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.637361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7661ef5b-e02a-4cc0-8680-9b65ede08a92-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-dsmqw\" (UID: \"7661ef5b-e02a-4cc0-8680-9b65ede08a92\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw"
Aug 01 00:39:22.637514 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.637410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clk6c\" (UniqueName: \"kubernetes.io/projected/7661ef5b-e02a-4cc0-8680-9b65ede08a92-kube-api-access-clk6c\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-dsmqw\" (UID: \"7661ef5b-e02a-4cc0-8680-9b65ede08a92\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw"
Aug 01 00:39:22.640243 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.640218    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7661ef5b-e02a-4cc0-8680-9b65ede08a92-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-dsmqw\" (UID: \"7661ef5b-e02a-4cc0-8680-9b65ede08a92\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw"
Aug 01 00:39:22.644553 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.644528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clk6c\" (UniqueName: \"kubernetes.io/projected/7661ef5b-e02a-4cc0-8680-9b65ede08a92-kube-api-access-clk6c\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-dsmqw\" (UID: \"7661ef5b-e02a-4cc0-8680-9b65ede08a92\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw"
Aug 01 00:39:22.734796 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.734756    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw"
Aug 01 00:39:22.882099 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:22.882073    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw"]
Aug 01 00:39:22.884401 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:39:22.884375    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7661ef5b_e02a_4cc0_8680_9b65ede08a92.slice/crio-55ccabc33ea756d9e17d3a8a518249199a056115607d85670ed32c51e9e15496 WatchSource:0}: Error finding container 55ccabc33ea756d9e17d3a8a518249199a056115607d85670ed32c51e9e15496: Status 404 returned error can't find the container with id 55ccabc33ea756d9e17d3a8a518249199a056115607d85670ed32c51e9e15496
Aug 01 00:39:23.014412 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:23.014369    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw" event={"ID":"7661ef5b-e02a-4cc0-8680-9b65ede08a92","Type":"ContainerStarted","Data":"28a5c82b7e9f7e4c5e48df730ef3d7c80bd3e3ff7cbc5cf83ac4463e78e3865f"}
Aug 01 00:39:23.014622 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:23.014441    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw" event={"ID":"7661ef5b-e02a-4cc0-8680-9b65ede08a92","Type":"ContainerStarted","Data":"55ccabc33ea756d9e17d3a8a518249199a056115607d85670ed32c51e9e15496"}
Aug 01 00:39:23.014622 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:23.014477    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw"
Aug 01 00:39:23.016621 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:23.016586    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" event={"ID":"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c","Type":"ContainerStarted","Data":"4e772036c0549cf032bf690f42c105ddcec5bda8f0899209c61005b2d5aa9f66"}
Aug 01 00:39:23.016796 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:23.016643    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" event={"ID":"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c","Type":"ContainerStarted","Data":"bb8a0d2997ea568f1cb4aa5fd4eac2c937d6e7eace47e860ca69084ba75f59c2"}
Aug 01 00:39:23.032109 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:23.032043    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw" podStartSLOduration=1.032020105 podStartE2EDuration="1.032020105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:39:23.030571495 +0000 UTC m=+3202.390462537" watchObservedRunningTime="2026-08-01 00:39:23.032020105 +0000 UTC m=+3202.391911147"
Aug 01 00:39:26.038918 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:26.038883    2578 generic.go:356] "Generic (PLEG): container finished" podID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerID="4e772036c0549cf032bf690f42c105ddcec5bda8f0899209c61005b2d5aa9f66" exitCode=0
Aug 01 00:39:26.039306 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:26.038959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" event={"ID":"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c","Type":"ContainerDied","Data":"4e772036c0549cf032bf690f42c105ddcec5bda8f0899209c61005b2d5aa9f66"}
Aug 01 00:39:27.046313 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:27.046270    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" event={"ID":"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c","Type":"ContainerStarted","Data":"12169692e5f8340c7a4ea4be08745c688d27d77214d4cc26d8f32884247c6fcd"}
Aug 01 00:39:27.068682 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:27.068626    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" podStartSLOduration=2.732080973 podStartE2EDuration="5.068607348s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.336526375s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:39:27.065098067 +0000 UTC m=+3206.424989124" watchObservedRunningTime="2026-08-01 00:39:27.068607348 +0000 UTC m=+3206.428498388"
Aug 01 00:39:27.475033 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:27.474992    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:32.474540 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:32.474505    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:39:32.476175 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:32.476140    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Aug 01 00:39:42.475149 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:42.475102    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Aug 01 00:39:52.475544 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:52.475454    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Aug 01 00:39:54.028072 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:39:54.028044    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw"
Aug 01 00:40:02.475614 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:40:02.475561    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Aug 01 00:40:12.474733 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:40:12.474659    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Aug 01 00:40:22.474708 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:40:22.474659    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Aug 01 00:40:32.474947 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:40:32.474902    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Aug 01 00:40:42.475149 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:40:42.475107    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Aug 01 00:40:52.475645 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:40:52.475600    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Aug 01 00:41:01.499695 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:01.499667    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:41:01.510150 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:01.510119    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:41:02.475800 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:02.475745    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Aug 01 00:41:12.476959 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:12.476910    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Aug 01 00:41:22.475310 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:22.475223    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Aug 01 00:41:32.485496 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:32.485461    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:41:32.493116 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:32.493091    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:41:37.962458 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:37.962421    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw"]
Aug 01 00:41:37.968281 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:37.968253    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"]
Aug 01 00:41:44.028100 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:44.028061    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw" podUID="7661ef5b-e02a-4cc0-8680-9b65ede08a92" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:41:52.984348 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:52.984280    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw" podUID="7661ef5b-e02a-4cc0-8680-9b65ede08a92" containerName="main" containerID="cri-o://28a5c82b7e9f7e4c5e48df730ef3d7c80bd3e3ff7cbc5cf83ac4463e78e3865f" gracePeriod=45
Aug 01 00:41:52.991008 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:52.990934    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="main" containerID="cri-o://12169692e5f8340c7a4ea4be08745c688d27d77214d4cc26d8f32884247c6fcd" gracePeriod=45
Aug 01 00:41:53.243550 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.243484    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw"
Aug 01 00:41:53.318369 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.318335    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clk6c\" (UniqueName: \"kubernetes.io/projected/7661ef5b-e02a-4cc0-8680-9b65ede08a92-kube-api-access-clk6c\") pod \"7661ef5b-e02a-4cc0-8680-9b65ede08a92\" (UID: \"7661ef5b-e02a-4cc0-8680-9b65ede08a92\") "
Aug 01 00:41:53.318569 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.318431    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7661ef5b-e02a-4cc0-8680-9b65ede08a92-tls-certs\") pod \"7661ef5b-e02a-4cc0-8680-9b65ede08a92\" (UID: \"7661ef5b-e02a-4cc0-8680-9b65ede08a92\") "
Aug 01 00:41:53.320825 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.320792    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7661ef5b-e02a-4cc0-8680-9b65ede08a92-kube-api-access-clk6c" (OuterVolumeSpecName: "kube-api-access-clk6c") pod "7661ef5b-e02a-4cc0-8680-9b65ede08a92" (UID: "7661ef5b-e02a-4cc0-8680-9b65ede08a92"). InnerVolumeSpecName "kube-api-access-clk6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:41:53.320950 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.320800    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7661ef5b-e02a-4cc0-8680-9b65ede08a92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7661ef5b-e02a-4cc0-8680-9b65ede08a92" (UID: "7661ef5b-e02a-4cc0-8680-9b65ede08a92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:41:53.419958 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.419921    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7661ef5b-e02a-4cc0-8680-9b65ede08a92-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:41:53.419958 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.419950    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clk6c\" (UniqueName: \"kubernetes.io/projected/7661ef5b-e02a-4cc0-8680-9b65ede08a92-kube-api-access-clk6c\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:41:53.486986 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.486936    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Aug 01 00:41:53.733319 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.733292    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:41:53.823783 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.823652    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-tmp-dir\") pod \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") "
Aug 01 00:41:53.823783 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.823705    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-tls-certs\") pod \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") "
Aug 01 00:41:53.823783 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.823771    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-model-cache\") pod \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") "
Aug 01 00:41:53.824078 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.823810    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-dshm\") pod \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") "
Aug 01 00:41:53.824078 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.823875    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-home\") pod \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") "
Aug 01 00:41:53.824078 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.823930    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-cabundle-cert\") pod \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") "
Aug 01 00:41:53.824078 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.823963    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm5ck\" (UniqueName: \"kubernetes.io/projected/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-kube-api-access-tm5ck\") pod \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") "
Aug 01 00:41:53.824078 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.823992    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-kserve-provision-location\") pod \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\" (UID: \"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c\") "
Aug 01 00:41:53.824354 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.824064    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" (UID: "2e829963-b5e4-44ca-b7b5-7de18eeb2b9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:41:53.824354 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.824276    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-model-cache\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:41:53.824354 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.824321    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" (UID: "2e829963-b5e4-44ca-b7b5-7de18eeb2b9c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:41:53.824534 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.824473    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-home" (OuterVolumeSpecName: "home") pod "2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" (UID: "2e829963-b5e4-44ca-b7b5-7de18eeb2b9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:41:53.826385 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.826354    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-kube-api-access-tm5ck" (OuterVolumeSpecName: "kube-api-access-tm5ck") pod "2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" (UID: "2e829963-b5e4-44ca-b7b5-7de18eeb2b9c"). InnerVolumeSpecName "kube-api-access-tm5ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:41:53.826611 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.826594    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" (UID: "2e829963-b5e4-44ca-b7b5-7de18eeb2b9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:41:53.826611 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.826599    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-dshm" (OuterVolumeSpecName: "dshm") pod "2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" (UID: "2e829963-b5e4-44ca-b7b5-7de18eeb2b9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:41:53.836171 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.836143    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" (UID: "2e829963-b5e4-44ca-b7b5-7de18eeb2b9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:41:53.850294 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.850269    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" (UID: "2e829963-b5e4-44ca-b7b5-7de18eeb2b9c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:41:53.919627 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.919600    2578 generic.go:356] "Generic (PLEG): container finished" podID="7661ef5b-e02a-4cc0-8680-9b65ede08a92" containerID="28a5c82b7e9f7e4c5e48df730ef3d7c80bd3e3ff7cbc5cf83ac4463e78e3865f" exitCode=0
Aug 01 00:41:53.919811 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.919646    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw"
Aug 01 00:41:53.919811 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.919687    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw" event={"ID":"7661ef5b-e02a-4cc0-8680-9b65ede08a92","Type":"ContainerDied","Data":"28a5c82b7e9f7e4c5e48df730ef3d7c80bd3e3ff7cbc5cf83ac4463e78e3865f"}
Aug 01 00:41:53.919811 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.919778    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw" event={"ID":"7661ef5b-e02a-4cc0-8680-9b65ede08a92","Type":"ContainerDied","Data":"55ccabc33ea756d9e17d3a8a518249199a056115607d85670ed32c51e9e15496"}
Aug 01 00:41:53.919811 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.919812    2578 scope.go:117] "RemoveContainer" containerID="28a5c82b7e9f7e4c5e48df730ef3d7c80bd3e3ff7cbc5cf83ac4463e78e3865f"
Aug 01 00:41:53.921457 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.921436    2578 generic.go:356] "Generic (PLEG): container finished" podID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerID="12169692e5f8340c7a4ea4be08745c688d27d77214d4cc26d8f32884247c6fcd" exitCode=0
Aug 01 00:41:53.921568 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.921491    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"
Aug 01 00:41:53.921627 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.921600    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" event={"ID":"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c","Type":"ContainerDied","Data":"12169692e5f8340c7a4ea4be08745c688d27d77214d4cc26d8f32884247c6fcd"}
Aug 01 00:41:53.921692 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.921643    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7" event={"ID":"2e829963-b5e4-44ca-b7b5-7de18eeb2b9c","Type":"ContainerDied","Data":"bb8a0d2997ea568f1cb4aa5fd4eac2c937d6e7eace47e860ca69084ba75f59c2"}
Aug 01 00:41:53.924787 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.924754    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:41:53.924787 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.924781    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-dshm\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:41:53.924951 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.924793    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-home\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:41:53.924951 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.924806    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-cabundle-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:41:53.924951 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.924821    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm5ck\" (UniqueName: \"kubernetes.io/projected/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-kube-api-access-tm5ck\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:41:53.924951 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.924835    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-kserve-provision-location\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:41:53.924951 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.924848    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c-tmp-dir\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:41:53.929781 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.929765    2578 scope.go:117] "RemoveContainer" containerID="28a5c82b7e9f7e4c5e48df730ef3d7c80bd3e3ff7cbc5cf83ac4463e78e3865f"
Aug 01 00:41:53.930033 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:41:53.930012    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28a5c82b7e9f7e4c5e48df730ef3d7c80bd3e3ff7cbc5cf83ac4463e78e3865f\": container with ID starting with 28a5c82b7e9f7e4c5e48df730ef3d7c80bd3e3ff7cbc5cf83ac4463e78e3865f not found: ID does not exist" containerID="28a5c82b7e9f7e4c5e48df730ef3d7c80bd3e3ff7cbc5cf83ac4463e78e3865f"
Aug 01 00:41:53.930116 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.930044    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a5c82b7e9f7e4c5e48df730ef3d7c80bd3e3ff7cbc5cf83ac4463e78e3865f"} err="failed to get container status \"28a5c82b7e9f7e4c5e48df730ef3d7c80bd3e3ff7cbc5cf83ac4463e78e3865f\": rpc error: code = NotFound desc = could not find container \"28a5c82b7e9f7e4c5e48df730ef3d7c80bd3e3ff7cbc5cf83ac4463e78e3865f\": container with ID starting with 28a5c82b7e9f7e4c5e48df730ef3d7c80bd3e3ff7cbc5cf83ac4463e78e3865f not found: ID does not exist"
Aug 01 00:41:53.930116 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.930070    2578 scope.go:117] "RemoveContainer" containerID="12169692e5f8340c7a4ea4be08745c688d27d77214d4cc26d8f32884247c6fcd"
Aug 01 00:41:53.939233 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.939214    2578 scope.go:117] "RemoveContainer" containerID="4e772036c0549cf032bf690f42c105ddcec5bda8f0899209c61005b2d5aa9f66"
Aug 01 00:41:53.941635 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.941611    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw"]
Aug 01 00:41:53.944364 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.944344    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-dsmqw"]
Aug 01 00:41:53.947912 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.947896    2578 scope.go:117] "RemoveContainer" containerID="12169692e5f8340c7a4ea4be08745c688d27d77214d4cc26d8f32884247c6fcd"
Aug 01 00:41:53.948200 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:41:53.948180    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12169692e5f8340c7a4ea4be08745c688d27d77214d4cc26d8f32884247c6fcd\": container with ID starting with 12169692e5f8340c7a4ea4be08745c688d27d77214d4cc26d8f32884247c6fcd not found: ID does not exist" containerID="12169692e5f8340c7a4ea4be08745c688d27d77214d4cc26d8f32884247c6fcd"
Aug 01 00:41:53.948257 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.948208    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12169692e5f8340c7a4ea4be08745c688d27d77214d4cc26d8f32884247c6fcd"} err="failed to get container status \"12169692e5f8340c7a4ea4be08745c688d27d77214d4cc26d8f32884247c6fcd\": rpc error: code = NotFound desc = could not find container \"12169692e5f8340c7a4ea4be08745c688d27d77214d4cc26d8f32884247c6fcd\": container with ID starting with 12169692e5f8340c7a4ea4be08745c688d27d77214d4cc26d8f32884247c6fcd not found: ID does not exist"
Aug 01 00:41:53.948257 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.948225    2578 scope.go:117] "RemoveContainer" containerID="4e772036c0549cf032bf690f42c105ddcec5bda8f0899209c61005b2d5aa9f66"
Aug 01 00:41:53.948486 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:41:53.948469    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e772036c0549cf032bf690f42c105ddcec5bda8f0899209c61005b2d5aa9f66\": container with ID starting with 4e772036c0549cf032bf690f42c105ddcec5bda8f0899209c61005b2d5aa9f66 not found: ID does not exist" containerID="4e772036c0549cf032bf690f42c105ddcec5bda8f0899209c61005b2d5aa9f66"
Aug 01 00:41:53.948532 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.948492    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e772036c0549cf032bf690f42c105ddcec5bda8f0899209c61005b2d5aa9f66"} err="failed to get container status \"4e772036c0549cf032bf690f42c105ddcec5bda8f0899209c61005b2d5aa9f66\": rpc error: code = NotFound desc = could not find container \"4e772036c0549cf032bf690f42c105ddcec5bda8f0899209c61005b2d5aa9f66\": container with ID starting with 4e772036c0549cf032bf690f42c105ddcec5bda8f0899209c61005b2d5aa9f66 not found: ID does not exist"
Aug 01 00:41:53.953810 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.953788    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"]
Aug 01 00:41:53.956800 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:53.956779    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64cbb688cd-dwtr7"]
Aug 01 00:41:55.184825 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:55.184792    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" path="/var/lib/kubelet/pods/2e829963-b5e4-44ca-b7b5-7de18eeb2b9c/volumes"
Aug 01 00:41:55.187416 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:41:55.185344    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7661ef5b-e02a-4cc0-8680-9b65ede08a92" path="/var/lib/kubelet/pods/7661ef5b-e02a-4cc0-8680-9b65ede08a92/volumes"
Aug 01 00:42:01.673729 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.673674    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"]
Aug 01 00:42:01.674286 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.674265    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="main"
Aug 01 00:42:01.674381 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.674289    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="main"
Aug 01 00:42:01.674381 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.674304    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="storage-initializer"
Aug 01 00:42:01.674381 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.674312    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="storage-initializer"
Aug 01 00:42:01.674381 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.674325    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7661ef5b-e02a-4cc0-8680-9b65ede08a92" containerName="main"
Aug 01 00:42:01.674381 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.674334    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7661ef5b-e02a-4cc0-8680-9b65ede08a92" containerName="main"
Aug 01 00:42:01.674634 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.674426    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7661ef5b-e02a-4cc0-8680-9b65ede08a92" containerName="main"
Aug 01 00:42:01.674634 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.674439    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e829963-b5e4-44ca-b7b5-7de18eeb2b9c" containerName="main"
Aug 01 00:42:01.677986 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.677964    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.679634 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.679608    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-rkqvb\""
Aug 01 00:42:01.679811 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.679609    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Aug 01 00:42:01.679811 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.679681    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:42:01.680089 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.680069    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Aug 01 00:42:01.680089 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.680079    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Aug 01 00:42:01.680264 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.680155    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Aug 01 00:42:01.688619 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.688595    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"]
Aug 01 00:42:01.798710 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.798672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-cabundle-cert\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.798934 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.798746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-home\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.798934 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.798802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-tmp-dir\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.798934 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.798829    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w8th\" (UniqueName: \"kubernetes.io/projected/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-kube-api-access-4w8th\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.798934 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.798855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-dshm\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.799140 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.798937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-tls-certs\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.799140 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.798968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-model-cache\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.799140 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.798986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-kserve-provision-location\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.900421 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.900379    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-cabundle-cert\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.900421 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.900426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-home\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.900695 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.900576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-tmp-dir\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.900695 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.900624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w8th\" (UniqueName: \"kubernetes.io/projected/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-kube-api-access-4w8th\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.900695 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.900656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-dshm\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.900906 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.900747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-tls-certs\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.900906 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.900775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-model-cache\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.900906 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.900797    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-kserve-provision-location\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.900906 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.900874    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-home\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.901121 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.900932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-tmp-dir\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.901121 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.901105    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-kserve-provision-location\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.901237 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.901166    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-model-cache\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.901514 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.901493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-cabundle-cert\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.903228 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.903206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-dshm\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.903352 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.903332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-tls-certs\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.912258 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.912223    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w8th\" (UniqueName: \"kubernetes.io/projected/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-kube-api-access-4w8th\") pod \"prestop-hook-test-kserve-65d966bbf4-rkwqn\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:01.912495 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.912473    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz"]
Aug 01 00:42:01.918145 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.918123    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz"
Aug 01 00:42:01.919906 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.919884    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-xcpr7\""
Aug 01 00:42:01.924221 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.924159    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz"]
Aug 01 00:42:01.991418 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:01.991383    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:02.102515 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:02.102471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e532f33-4457-46c9-a065-bcad8ea91e20-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz\" (UID: \"7e532f33-4457-46c9-a065-bcad8ea91e20\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz"
Aug 01 00:42:02.102685 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:02.102535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gltj\" (UniqueName: \"kubernetes.io/projected/7e532f33-4457-46c9-a065-bcad8ea91e20-kube-api-access-4gltj\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz\" (UID: \"7e532f33-4457-46c9-a065-bcad8ea91e20\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz"
Aug 01 00:42:02.139906 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:02.139876    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"]
Aug 01 00:42:02.142489 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:42:02.142461    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8c81f4_af23_4a78_a4eb_2af6444db7f7.slice/crio-33f8a7a98ae1eb0e1c8b6421433ab5846e641576f3b44b09510e803e6a478aee WatchSource:0}: Error finding container 33f8a7a98ae1eb0e1c8b6421433ab5846e641576f3b44b09510e803e6a478aee: Status 404 returned error can't find the container with id 33f8a7a98ae1eb0e1c8b6421433ab5846e641576f3b44b09510e803e6a478aee
Aug 01 00:42:02.203864 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:02.203840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e532f33-4457-46c9-a065-bcad8ea91e20-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz\" (UID: \"7e532f33-4457-46c9-a065-bcad8ea91e20\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz"
Aug 01 00:42:02.204019 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:02.203876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gltj\" (UniqueName: \"kubernetes.io/projected/7e532f33-4457-46c9-a065-bcad8ea91e20-kube-api-access-4gltj\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz\" (UID: \"7e532f33-4457-46c9-a065-bcad8ea91e20\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz"
Aug 01 00:42:02.206355 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:02.206336    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e532f33-4457-46c9-a065-bcad8ea91e20-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz\" (UID: \"7e532f33-4457-46c9-a065-bcad8ea91e20\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz"
Aug 01 00:42:02.210863 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:02.210840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gltj\" (UniqueName: \"kubernetes.io/projected/7e532f33-4457-46c9-a065-bcad8ea91e20-kube-api-access-4gltj\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz\" (UID: \"7e532f33-4457-46c9-a065-bcad8ea91e20\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz"
Aug 01 00:42:02.234938 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:02.234908    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz"
Aug 01 00:42:02.378666 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:02.378636    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz"]
Aug 01 00:42:02.380895 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:42:02.380872    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e532f33_4457_46c9_a065_bcad8ea91e20.slice/crio-240da0111ae788b09d1cf97069d1fdd79346dcb2ef13f0b804ef75ca372f46a8 WatchSource:0}: Error finding container 240da0111ae788b09d1cf97069d1fdd79346dcb2ef13f0b804ef75ca372f46a8: Status 404 returned error can't find the container with id 240da0111ae788b09d1cf97069d1fdd79346dcb2ef13f0b804ef75ca372f46a8
Aug 01 00:42:02.975924 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:02.975884    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz" event={"ID":"7e532f33-4457-46c9-a065-bcad8ea91e20","Type":"ContainerStarted","Data":"d9ee2913ba03af9a94259897bc1e9996e858e23f09cca3c5e2d4ab0376c246b7"}
Aug 01 00:42:02.976485 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:02.975958    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz" event={"ID":"7e532f33-4457-46c9-a065-bcad8ea91e20","Type":"ContainerStarted","Data":"240da0111ae788b09d1cf97069d1fdd79346dcb2ef13f0b804ef75ca372f46a8"}
Aug 01 00:42:02.976485 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:02.975997    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz"
Aug 01 00:42:02.977636 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:02.977611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" event={"ID":"3b8c81f4-af23-4a78-a4eb-2af6444db7f7","Type":"ContainerStarted","Data":"881c8826e5d4f1fcd26058d8c2142189120d4c6936bdadac44ab4bae5701b4ab"}
Aug 01 00:42:02.977780 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:02.977660    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" event={"ID":"3b8c81f4-af23-4a78-a4eb-2af6444db7f7","Type":"ContainerStarted","Data":"33f8a7a98ae1eb0e1c8b6421433ab5846e641576f3b44b09510e803e6a478aee"}
Aug 01 00:42:03.001126 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:03.001075    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz" podStartSLOduration=2.001059024 podStartE2EDuration="2.001059024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:42:02.998094105 +0000 UTC m=+3362.357985158" watchObservedRunningTime="2026-08-01 00:42:03.001059024 +0000 UTC m=+3362.360950066"
Aug 01 00:42:04.989972 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:04.989935    2578 generic.go:356] "Generic (PLEG): container finished" podID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerID="881c8826e5d4f1fcd26058d8c2142189120d4c6936bdadac44ab4bae5701b4ab" exitCode=0
Aug 01 00:42:04.990395 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:04.990032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" event={"ID":"3b8c81f4-af23-4a78-a4eb-2af6444db7f7","Type":"ContainerDied","Data":"881c8826e5d4f1fcd26058d8c2142189120d4c6936bdadac44ab4bae5701b4ab"}
Aug 01 00:42:05.998836 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:05.998802    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" event={"ID":"3b8c81f4-af23-4a78-a4eb-2af6444db7f7","Type":"ContainerStarted","Data":"111636de55dc0eeeb9e7e532797300a1c6b50dcdfacf4b2743de5e151ba5c129"}
Aug 01 00:42:06.018017 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:06.017951    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" podStartSLOduration=2.726190522 podStartE2EDuration="5.017929831s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.291739309s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:42:06.016587126 +0000 UTC m=+3365.376478166" watchObservedRunningTime="2026-08-01 00:42:06.017929831 +0000 UTC m=+3365.377820876"
Aug 01 00:42:06.992252 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:06.992214    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:11.991672 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:11.991627    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:42:11.993548 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:11.993512    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Aug 01 00:42:21.992183 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:21.992133    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Aug 01 00:42:31.992869 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:31.992822    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Aug 01 00:42:33.985175 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:33.985144    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz"
Aug 01 00:42:41.992894 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:41.992846    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Aug 01 00:42:51.992006 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:42:51.991906    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Aug 01 00:43:01.992097 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:43:01.992043    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Aug 01 00:43:11.992340 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:43:11.992290    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Aug 01 00:43:21.992921 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:43:21.992873    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Aug 01 00:43:31.992324 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:43:31.992270    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Aug 01 00:43:41.992214 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:43:41.992159    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Aug 01 00:43:51.992798 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:43:51.992749    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Aug 01 00:44:01.992373 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:01.992325    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Aug 01 00:44:12.003478 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:12.003442    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:44:12.012154 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:12.012120    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:44:13.421014 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:13.420974    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"]
Aug 01 00:44:28.790836 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:28.790707    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="main" containerID="cri-o://111636de55dc0eeeb9e7e532797300a1c6b50dcdfacf4b2743de5e151ba5c129" gracePeriod=45
Aug 01 00:44:29.004508 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.004460    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Aug 01 00:44:29.542397 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.542372    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:44:29.595330 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.595241    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-kserve-provision-location\") pod \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") "
Aug 01 00:44:29.595330 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.595285    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-cabundle-cert\") pod \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") "
Aug 01 00:44:29.595330 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.595305    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-home\") pod \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") "
Aug 01 00:44:29.595330 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.595325    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-tmp-dir\") pod \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") "
Aug 01 00:44:29.595690 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.595378    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-dshm\") pod \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") "
Aug 01 00:44:29.595690 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.595435    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w8th\" (UniqueName: \"kubernetes.io/projected/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-kube-api-access-4w8th\") pod \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") "
Aug 01 00:44:29.595690 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.595461    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-tls-certs\") pod \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") "
Aug 01 00:44:29.595690 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.595477    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-model-cache\") pod \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\" (UID: \"3b8c81f4-af23-4a78-a4eb-2af6444db7f7\") "
Aug 01 00:44:29.595942 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.595701    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3b8c81f4-af23-4a78-a4eb-2af6444db7f7" (UID: "3b8c81f4-af23-4a78-a4eb-2af6444db7f7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:44:29.595942 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.595924    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b8c81f4-af23-4a78-a4eb-2af6444db7f7" (UID: "3b8c81f4-af23-4a78-a4eb-2af6444db7f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:44:29.596089 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.596059    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-home" (OuterVolumeSpecName: "home") pod "3b8c81f4-af23-4a78-a4eb-2af6444db7f7" (UID: "3b8c81f4-af23-4a78-a4eb-2af6444db7f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:44:29.598355 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.598311    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-dshm" (OuterVolumeSpecName: "dshm") pod "3b8c81f4-af23-4a78-a4eb-2af6444db7f7" (UID: "3b8c81f4-af23-4a78-a4eb-2af6444db7f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:44:29.598468 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.598353    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b8c81f4-af23-4a78-a4eb-2af6444db7f7" (UID: "3b8c81f4-af23-4a78-a4eb-2af6444db7f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:44:29.598468 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.598410    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-kube-api-access-4w8th" (OuterVolumeSpecName: "kube-api-access-4w8th") pod "3b8c81f4-af23-4a78-a4eb-2af6444db7f7" (UID: "3b8c81f4-af23-4a78-a4eb-2af6444db7f7"). InnerVolumeSpecName "kube-api-access-4w8th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:44:29.613359 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.613328    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b8c81f4-af23-4a78-a4eb-2af6444db7f7" (UID: "3b8c81f4-af23-4a78-a4eb-2af6444db7f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:44:29.621998 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.621976    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3b8c81f4-af23-4a78-a4eb-2af6444db7f7" (UID: "3b8c81f4-af23-4a78-a4eb-2af6444db7f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:44:29.696522 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.696486    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-dshm\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:44:29.696522 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.696518    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w8th\" (UniqueName: \"kubernetes.io/projected/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-kube-api-access-4w8th\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:44:29.696522 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.696529    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:44:29.696780 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.696539    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-model-cache\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:44:29.696780 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.696548    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-kserve-provision-location\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:44:29.696780 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.696555    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-cabundle-cert\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:44:29.696780 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.696563    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-home\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:44:29.696780 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.696570    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8c81f4-af23-4a78-a4eb-2af6444db7f7-tmp-dir\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:44:29.856814 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.856708    2578 generic.go:356] "Generic (PLEG): container finished" podID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerID="111636de55dc0eeeb9e7e532797300a1c6b50dcdfacf4b2743de5e151ba5c129" exitCode=0
Aug 01 00:44:29.856814 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.856777    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"
Aug 01 00:44:29.856814 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.856806    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" event={"ID":"3b8c81f4-af23-4a78-a4eb-2af6444db7f7","Type":"ContainerDied","Data":"111636de55dc0eeeb9e7e532797300a1c6b50dcdfacf4b2743de5e151ba5c129"}
Aug 01 00:44:29.857358 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.856851    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn" event={"ID":"3b8c81f4-af23-4a78-a4eb-2af6444db7f7","Type":"ContainerDied","Data":"33f8a7a98ae1eb0e1c8b6421433ab5846e641576f3b44b09510e803e6a478aee"}
Aug 01 00:44:29.857358 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.856873    2578 scope.go:117] "RemoveContainer" containerID="111636de55dc0eeeb9e7e532797300a1c6b50dcdfacf4b2743de5e151ba5c129"
Aug 01 00:44:29.867378 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.867355    2578 scope.go:117] "RemoveContainer" containerID="881c8826e5d4f1fcd26058d8c2142189120d4c6936bdadac44ab4bae5701b4ab"
Aug 01 00:44:29.881614 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.881588    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"]
Aug 01 00:44:29.883058 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.883021    2578 scope.go:117] "RemoveContainer" containerID="111636de55dc0eeeb9e7e532797300a1c6b50dcdfacf4b2743de5e151ba5c129"
Aug 01 00:44:29.883469 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:44:29.883448    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111636de55dc0eeeb9e7e532797300a1c6b50dcdfacf4b2743de5e151ba5c129\": container with ID starting with 111636de55dc0eeeb9e7e532797300a1c6b50dcdfacf4b2743de5e151ba5c129 not found: ID does not exist" containerID="111636de55dc0eeeb9e7e532797300a1c6b50dcdfacf4b2743de5e151ba5c129"
Aug 01 00:44:29.883548 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.883475    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111636de55dc0eeeb9e7e532797300a1c6b50dcdfacf4b2743de5e151ba5c129"} err="failed to get container status \"111636de55dc0eeeb9e7e532797300a1c6b50dcdfacf4b2743de5e151ba5c129\": rpc error: code = NotFound desc = could not find container \"111636de55dc0eeeb9e7e532797300a1c6b50dcdfacf4b2743de5e151ba5c129\": container with ID starting with 111636de55dc0eeeb9e7e532797300a1c6b50dcdfacf4b2743de5e151ba5c129 not found: ID does not exist"
Aug 01 00:44:29.883548 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.883493    2578 scope.go:117] "RemoveContainer" containerID="881c8826e5d4f1fcd26058d8c2142189120d4c6936bdadac44ab4bae5701b4ab"
Aug 01 00:44:29.883816 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:44:29.883798    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"881c8826e5d4f1fcd26058d8c2142189120d4c6936bdadac44ab4bae5701b4ab\": container with ID starting with 881c8826e5d4f1fcd26058d8c2142189120d4c6936bdadac44ab4bae5701b4ab not found: ID does not exist" containerID="881c8826e5d4f1fcd26058d8c2142189120d4c6936bdadac44ab4bae5701b4ab"
Aug 01 00:44:29.883882 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.883821    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"881c8826e5d4f1fcd26058d8c2142189120d4c6936bdadac44ab4bae5701b4ab"} err="failed to get container status \"881c8826e5d4f1fcd26058d8c2142189120d4c6936bdadac44ab4bae5701b4ab\": rpc error: code = NotFound desc = could not find container \"881c8826e5d4f1fcd26058d8c2142189120d4c6936bdadac44ab4bae5701b4ab\": container with ID starting with 881c8826e5d4f1fcd26058d8c2142189120d4c6936bdadac44ab4bae5701b4ab not found: ID does not exist"
Aug 01 00:44:29.885597 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:29.885578    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-rkwqn"]
Aug 01 00:44:31.186916 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:31.186887    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" path="/var/lib/kubelet/pods/3b8c81f4-af23-4a78-a4eb-2af6444db7f7/volumes"
Aug 01 00:44:32.224599 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:32.224563    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz"]
Aug 01 00:44:33.984530 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:33.984487    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz" podUID="7e532f33-4457-46c9-a065-bcad8ea91e20" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:44:43.985187 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:43.985150    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz" podUID="7e532f33-4457-46c9-a065-bcad8ea91e20" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:44:47.249114 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:47.249074    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz" podUID="7e532f33-4457-46c9-a065-bcad8ea91e20" containerName="main" containerID="cri-o://d9ee2913ba03af9a94259897bc1e9996e858e23f09cca3c5e2d4ab0376c246b7" gracePeriod=45
Aug 01 00:44:47.499510 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:47.499446    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz"
Aug 01 00:44:47.660229 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:47.660190    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gltj\" (UniqueName: \"kubernetes.io/projected/7e532f33-4457-46c9-a065-bcad8ea91e20-kube-api-access-4gltj\") pod \"7e532f33-4457-46c9-a065-bcad8ea91e20\" (UID: \"7e532f33-4457-46c9-a065-bcad8ea91e20\") "
Aug 01 00:44:47.660437 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:47.660292    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e532f33-4457-46c9-a065-bcad8ea91e20-tls-certs\") pod \"7e532f33-4457-46c9-a065-bcad8ea91e20\" (UID: \"7e532f33-4457-46c9-a065-bcad8ea91e20\") "
Aug 01 00:44:47.662515 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:47.662490    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e532f33-4457-46c9-a065-bcad8ea91e20-kube-api-access-4gltj" (OuterVolumeSpecName: "kube-api-access-4gltj") pod "7e532f33-4457-46c9-a065-bcad8ea91e20" (UID: "7e532f33-4457-46c9-a065-bcad8ea91e20"). InnerVolumeSpecName "kube-api-access-4gltj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:44:47.662614 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:47.662528    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e532f33-4457-46c9-a065-bcad8ea91e20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e532f33-4457-46c9-a065-bcad8ea91e20" (UID: "7e532f33-4457-46c9-a065-bcad8ea91e20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:44:47.761892 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:47.761808    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e532f33-4457-46c9-a065-bcad8ea91e20-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:44:47.761892 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:47.761837    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gltj\" (UniqueName: \"kubernetes.io/projected/7e532f33-4457-46c9-a065-bcad8ea91e20-kube-api-access-4gltj\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:44:47.960980 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:47.960940    2578 generic.go:356] "Generic (PLEG): container finished" podID="7e532f33-4457-46c9-a065-bcad8ea91e20" containerID="d9ee2913ba03af9a94259897bc1e9996e858e23f09cca3c5e2d4ab0376c246b7" exitCode=0
Aug 01 00:44:47.961149 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:47.960988    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz"
Aug 01 00:44:47.961149 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:47.961034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz" event={"ID":"7e532f33-4457-46c9-a065-bcad8ea91e20","Type":"ContainerDied","Data":"d9ee2913ba03af9a94259897bc1e9996e858e23f09cca3c5e2d4ab0376c246b7"}
Aug 01 00:44:47.961149 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:47.961093    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz" event={"ID":"7e532f33-4457-46c9-a065-bcad8ea91e20","Type":"ContainerDied","Data":"240da0111ae788b09d1cf97069d1fdd79346dcb2ef13f0b804ef75ca372f46a8"}
Aug 01 00:44:47.961149 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:47.961121    2578 scope.go:117] "RemoveContainer" containerID="d9ee2913ba03af9a94259897bc1e9996e858e23f09cca3c5e2d4ab0376c246b7"
Aug 01 00:44:47.971582 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:47.971560    2578 scope.go:117] "RemoveContainer" containerID="d9ee2913ba03af9a94259897bc1e9996e858e23f09cca3c5e2d4ab0376c246b7"
Aug 01 00:44:47.971858 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:44:47.971835    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ee2913ba03af9a94259897bc1e9996e858e23f09cca3c5e2d4ab0376c246b7\": container with ID starting with d9ee2913ba03af9a94259897bc1e9996e858e23f09cca3c5e2d4ab0376c246b7 not found: ID does not exist" containerID="d9ee2913ba03af9a94259897bc1e9996e858e23f09cca3c5e2d4ab0376c246b7"
Aug 01 00:44:47.971940 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:47.971870    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ee2913ba03af9a94259897bc1e9996e858e23f09cca3c5e2d4ab0376c246b7"} err="failed to get container status \"d9ee2913ba03af9a94259897bc1e9996e858e23f09cca3c5e2d4ab0376c246b7\": rpc error: code = NotFound desc = could not find container \"d9ee2913ba03af9a94259897bc1e9996e858e23f09cca3c5e2d4ab0376c246b7\": container with ID starting with d9ee2913ba03af9a94259897bc1e9996e858e23f09cca3c5e2d4ab0376c246b7 not found: ID does not exist"
Aug 01 00:44:47.981776 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:47.981753    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz"]
Aug 01 00:44:47.985516 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:47.985490    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-vlnxz"]
Aug 01 00:44:49.185180 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:44:49.185147    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e532f33-4457-46c9-a065-bcad8ea91e20" path="/var/lib/kubelet/pods/7e532f33-4457-46c9-a065-bcad8ea91e20/volumes"
Aug 01 00:45:44.557987 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.557953    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"]
Aug 01 00:45:44.558683 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.558659    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e532f33-4457-46c9-a065-bcad8ea91e20" containerName="main"
Aug 01 00:45:44.558812 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.558686    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e532f33-4457-46c9-a065-bcad8ea91e20" containerName="main"
Aug 01 00:45:44.558812 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.558700    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="main"
Aug 01 00:45:44.558812 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.558708    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="main"
Aug 01 00:45:44.558812 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.558747    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="storage-initializer"
Aug 01 00:45:44.558812 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.558755    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="storage-initializer"
Aug 01 00:45:44.559073 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.558885    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e532f33-4457-46c9-a065-bcad8ea91e20" containerName="main"
Aug 01 00:45:44.559073 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.558897    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b8c81f4-af23-4a78-a4eb-2af6444db7f7" containerName="main"
Aug 01 00:45:44.562055 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.562034    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.563889 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.563867    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Aug 01 00:45:44.564225 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.564204    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Aug 01 00:45:44.564319 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.564228    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-mfwrn\""
Aug 01 00:45:44.564319 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.564240    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Aug 01 00:45:44.570477 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.569692    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"]
Aug 01 00:45:44.753109 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.753067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gskj5\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.753299 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.753128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gskj5\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.753299 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.753180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn2m7\" (UniqueName: \"kubernetes.io/projected/9b4d3515-8945-413d-a60b-fb089cd06e7d-kube-api-access-wn2m7\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gskj5\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.753299 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.753259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4d3515-8945-413d-a60b-fb089cd06e7d-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gskj5\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.753299 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.753286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gskj5\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.753493 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.753346    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gskj5\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.808523 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.808372    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng"]
Aug 01 00:45:44.812310 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.812287    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng"
Aug 01 00:45:44.813999 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.813970    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-t97rv\""
Aug 01 00:45:44.819880 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.819852    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng"]
Aug 01 00:45:44.854574 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.854537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gskj5\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.854798 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.854582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn2m7\" (UniqueName: \"kubernetes.io/projected/9b4d3515-8945-413d-a60b-fb089cd06e7d-kube-api-access-wn2m7\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gskj5\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.854798 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.854625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4d3515-8945-413d-a60b-fb089cd06e7d-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gskj5\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.854798 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.854653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gskj5\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.854798 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.854674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gskj5\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.854798 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.854709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gskj5\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.855102 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.855046    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gskj5\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.855102 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.855074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gskj5\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.855186 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.855145    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gskj5\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.857210 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.857183    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gskj5\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.857398 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.857382    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4d3515-8945-413d-a60b-fb089cd06e7d-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gskj5\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.861846 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.861823    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn2m7\" (UniqueName: \"kubernetes.io/projected/9b4d3515-8945-413d-a60b-fb089cd06e7d-kube-api-access-wn2m7\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gskj5\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.874702 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.874678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:44.956259 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.956227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7694cb77-6106-4868-8ed8-58b2bb10eae2-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng\" (UID: \"7694cb77-6106-4868-8ed8-58b2bb10eae2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng"
Aug 01 00:45:44.956438 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:44.956289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xktnb\" (UniqueName: \"kubernetes.io/projected/7694cb77-6106-4868-8ed8-58b2bb10eae2-kube-api-access-xktnb\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng\" (UID: \"7694cb77-6106-4868-8ed8-58b2bb10eae2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng"
Aug 01 00:45:45.016612 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:45.016443    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"]
Aug 01 00:45:45.019452 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:45:45.019424    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4d3515_8945_413d_a60b_fb089cd06e7d.slice/crio-aa19cd20044d15c6e866b7728280556d06a446079e403160a0dc66d025a3c5e0 WatchSource:0}: Error finding container aa19cd20044d15c6e866b7728280556d06a446079e403160a0dc66d025a3c5e0: Status 404 returned error can't find the container with id aa19cd20044d15c6e866b7728280556d06a446079e403160a0dc66d025a3c5e0
Aug 01 00:45:45.057740 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:45.057690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xktnb\" (UniqueName: \"kubernetes.io/projected/7694cb77-6106-4868-8ed8-58b2bb10eae2-kube-api-access-xktnb\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng\" (UID: \"7694cb77-6106-4868-8ed8-58b2bb10eae2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng"
Aug 01 00:45:45.057880 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:45.057811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7694cb77-6106-4868-8ed8-58b2bb10eae2-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng\" (UID: \"7694cb77-6106-4868-8ed8-58b2bb10eae2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng"
Aug 01 00:45:45.060480 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:45.060420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7694cb77-6106-4868-8ed8-58b2bb10eae2-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng\" (UID: \"7694cb77-6106-4868-8ed8-58b2bb10eae2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng"
Aug 01 00:45:45.065180 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:45.065155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xktnb\" (UniqueName: \"kubernetes.io/projected/7694cb77-6106-4868-8ed8-58b2bb10eae2-kube-api-access-xktnb\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng\" (UID: \"7694cb77-6106-4868-8ed8-58b2bb10eae2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng"
Aug 01 00:45:45.124273 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:45.124233    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng"
Aug 01 00:45:45.263108 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:45.263079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5" event={"ID":"9b4d3515-8945-413d-a60b-fb089cd06e7d","Type":"ContainerStarted","Data":"e9b65ff5c3596e0ce7acbac63cc18fe1b97046b8fe4ed161525d32966a030753"}
Aug 01 00:45:45.263282 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:45.263132    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5" event={"ID":"9b4d3515-8945-413d-a60b-fb089cd06e7d","Type":"ContainerStarted","Data":"aa19cd20044d15c6e866b7728280556d06a446079e403160a0dc66d025a3c5e0"}
Aug 01 00:45:45.265240 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:45.265215    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng"]
Aug 01 00:45:45.267225 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:45:45.267203    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7694cb77_6106_4868_8ed8_58b2bb10eae2.slice/crio-58ace1e14c3596ebaece24d94ed20f27a1dce40407079ad47624d827b381a527 WatchSource:0}: Error finding container 58ace1e14c3596ebaece24d94ed20f27a1dce40407079ad47624d827b381a527: Status 404 returned error can't find the container with id 58ace1e14c3596ebaece24d94ed20f27a1dce40407079ad47624d827b381a527
Aug 01 00:45:45.279267 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:45.279228    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5" podStartSLOduration=1.279212812 podStartE2EDuration="1.279212812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:45:45.277910463 +0000 UTC m=+3584.637801504" watchObservedRunningTime="2026-08-01 00:45:45.279212812 +0000 UTC m=+3584.639103852"
Aug 01 00:45:45.875274 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:45.875243    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:46.269708 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:46.269677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng" event={"ID":"7694cb77-6106-4868-8ed8-58b2bb10eae2","Type":"ContainerStarted","Data":"742e0e69fe4dd8c7210e489b67e81d64bc7aa3215ff940f36e4f85d95bf06b50"}
Aug 01 00:45:46.269939 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:46.269869    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng"
Aug 01 00:45:46.269939 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:46.269904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng" event={"ID":"7694cb77-6106-4868-8ed8-58b2bb10eae2","Type":"ContainerStarted","Data":"58ace1e14c3596ebaece24d94ed20f27a1dce40407079ad47624d827b381a527"}
Aug 01 00:45:46.285073 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:46.285023    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng" podStartSLOduration=2.285007318 podStartE2EDuration="2.285007318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:45:46.283542268 +0000 UTC m=+3585.643433301" watchObservedRunningTime="2026-08-01 00:45:46.285007318 +0000 UTC m=+3585.644898359"
Aug 01 00:45:54.875951 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:54.875854    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:54.888442 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:54.888411    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:45:55.333737 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:45:55.333696    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:46:01.536737 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:01.536685    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:46:01.548497 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:01.548476    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdzrq_030ae37c-094e-4a10-bea2-8e2b8bcc6d5e/ovn-acl-logging/0.log"
Aug 01 00:46:17.276746 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:17.276698    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng"
Aug 01 00:46:29.544497 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:29.544461    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"]
Aug 01 00:46:29.554610 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:46:29.554570    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:46:29.554610 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:46:29.554610 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Aug 01 00:46:29.554610 ip-10-0-140-3 kubenswrapper[2578]:  > containerID="e9b65ff5c3596e0ce7acbac63cc18fe1b97046b8fe4ed161525d32966a030753" cmd=["/bin/sleep","15"]
Aug 01 00:46:29.554930 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:29.554630    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5" message="" err=<
Aug 01 00:46:29.554930 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:46:29.554930 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Aug 01 00:46:29.554930 ip-10-0-140-3 kubenswrapper[2578]:  >
Aug 01 00:46:29.554930 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:46:29.554643    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:46:29.554930 ip-10-0-140-3 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:46:29.554930 ip-10-0-140-3 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Aug 01 00:46:29.554930 ip-10-0-140-3 kubenswrapper[2578]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5" podUID="9b4d3515-8945-413d-a60b-fb089cd06e7d" containerName="main" containerID="cri-o://e9b65ff5c3596e0ce7acbac63cc18fe1b97046b8fe4ed161525d32966a030753"
Aug 01 00:46:29.554930 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:29.554694    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5" podUID="9b4d3515-8945-413d-a60b-fb089cd06e7d" containerName="main" containerID="cri-o://e9b65ff5c3596e0ce7acbac63cc18fe1b97046b8fe4ed161525d32966a030753" gracePeriod=60
Aug 01 00:46:29.912142 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:29.912111    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:46:30.049003 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.048966    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-home\") pod \"9b4d3515-8945-413d-a60b-fb089cd06e7d\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") "
Aug 01 00:46:30.049176 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.049034    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4d3515-8945-413d-a60b-fb089cd06e7d-tls-certs\") pod \"9b4d3515-8945-413d-a60b-fb089cd06e7d\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") "
Aug 01 00:46:30.049176 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.049069    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-tmp-dir\") pod \"9b4d3515-8945-413d-a60b-fb089cd06e7d\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") "
Aug 01 00:46:30.049176 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.049092    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-dshm\") pod \"9b4d3515-8945-413d-a60b-fb089cd06e7d\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") "
Aug 01 00:46:30.049176 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.049114    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-model-cache\") pod \"9b4d3515-8945-413d-a60b-fb089cd06e7d\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") "
Aug 01 00:46:30.049176 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.049136    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wn2m7\" (UniqueName: \"kubernetes.io/projected/9b4d3515-8945-413d-a60b-fb089cd06e7d-kube-api-access-wn2m7\") pod \"9b4d3515-8945-413d-a60b-fb089cd06e7d\" (UID: \"9b4d3515-8945-413d-a60b-fb089cd06e7d\") "
Aug 01 00:46:30.049435 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.049294    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-home" (OuterVolumeSpecName: "home") pod "9b4d3515-8945-413d-a60b-fb089cd06e7d" (UID: "9b4d3515-8945-413d-a60b-fb089cd06e7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:30.049435 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.049315    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b4d3515-8945-413d-a60b-fb089cd06e7d" (UID: "9b4d3515-8945-413d-a60b-fb089cd06e7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:30.049435 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.049407    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b4d3515-8945-413d-a60b-fb089cd06e7d" (UID: "9b4d3515-8945-413d-a60b-fb089cd06e7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:30.049435 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.049428    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-home\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:46:30.049618 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.049439    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-tmp-dir\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:46:30.051551 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.051529    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-dshm" (OuterVolumeSpecName: "dshm") pod "9b4d3515-8945-413d-a60b-fb089cd06e7d" (UID: "9b4d3515-8945-413d-a60b-fb089cd06e7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:30.051655 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.051589    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4d3515-8945-413d-a60b-fb089cd06e7d-kube-api-access-wn2m7" (OuterVolumeSpecName: "kube-api-access-wn2m7") pod "9b4d3515-8945-413d-a60b-fb089cd06e7d" (UID: "9b4d3515-8945-413d-a60b-fb089cd06e7d"). InnerVolumeSpecName "kube-api-access-wn2m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:46:30.051655 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.051634    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b4d3515-8945-413d-a60b-fb089cd06e7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b4d3515-8945-413d-a60b-fb089cd06e7d" (UID: "9b4d3515-8945-413d-a60b-fb089cd06e7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:46:30.150296 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.150258    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4d3515-8945-413d-a60b-fb089cd06e7d-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:46:30.150296 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.150293    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-dshm\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:46:30.150491 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.150302    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b4d3515-8945-413d-a60b-fb089cd06e7d-model-cache\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:46:30.150491 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.150311    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn2m7\" (UniqueName: \"kubernetes.io/projected/9b4d3515-8945-413d-a60b-fb089cd06e7d-kube-api-access-wn2m7\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:46:30.515467 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.515378    2578 generic.go:356] "Generic (PLEG): container finished" podID="9b4d3515-8945-413d-a60b-fb089cd06e7d" containerID="e9b65ff5c3596e0ce7acbac63cc18fe1b97046b8fe4ed161525d32966a030753" exitCode=0
Aug 01 00:46:30.515467 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.515436    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"
Aug 01 00:46:30.515467 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.515457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5" event={"ID":"9b4d3515-8945-413d-a60b-fb089cd06e7d","Type":"ContainerDied","Data":"e9b65ff5c3596e0ce7acbac63cc18fe1b97046b8fe4ed161525d32966a030753"}
Aug 01 00:46:30.515754 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.515540    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5" event={"ID":"9b4d3515-8945-413d-a60b-fb089cd06e7d","Type":"ContainerDied","Data":"aa19cd20044d15c6e866b7728280556d06a446079e403160a0dc66d025a3c5e0"}
Aug 01 00:46:30.515754 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.515568    2578 scope.go:117] "RemoveContainer" containerID="e9b65ff5c3596e0ce7acbac63cc18fe1b97046b8fe4ed161525d32966a030753"
Aug 01 00:46:30.525773 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.525753    2578 scope.go:117] "RemoveContainer" containerID="e9b65ff5c3596e0ce7acbac63cc18fe1b97046b8fe4ed161525d32966a030753"
Aug 01 00:46:30.526029 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:46:30.526011    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b65ff5c3596e0ce7acbac63cc18fe1b97046b8fe4ed161525d32966a030753\": container with ID starting with e9b65ff5c3596e0ce7acbac63cc18fe1b97046b8fe4ed161525d32966a030753 not found: ID does not exist" containerID="e9b65ff5c3596e0ce7acbac63cc18fe1b97046b8fe4ed161525d32966a030753"
Aug 01 00:46:30.526071 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.526039    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b65ff5c3596e0ce7acbac63cc18fe1b97046b8fe4ed161525d32966a030753"} err="failed to get container status \"e9b65ff5c3596e0ce7acbac63cc18fe1b97046b8fe4ed161525d32966a030753\": rpc error: code = NotFound desc = could not find container \"e9b65ff5c3596e0ce7acbac63cc18fe1b97046b8fe4ed161525d32966a030753\": container with ID starting with e9b65ff5c3596e0ce7acbac63cc18fe1b97046b8fe4ed161525d32966a030753 not found: ID does not exist"
Aug 01 00:46:30.535982 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.535962    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"]
Aug 01 00:46:30.539755 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:30.539733    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gskj5"]
Aug 01 00:46:31.185358 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:31.185327    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b4d3515-8945-413d-a60b-fb089cd06e7d" path="/var/lib/kubelet/pods/9b4d3515-8945-413d-a60b-fb089cd06e7d/volumes"
Aug 01 00:46:41.657073 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:41.657033    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng"]
Aug 01 00:46:47.277516 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:47.277479    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng" podUID="7694cb77-6106-4868-8ed8-58b2bb10eae2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:46:56.678453 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:56.678400    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng" podUID="7694cb77-6106-4868-8ed8-58b2bb10eae2" containerName="main" containerID="cri-o://742e0e69fe4dd8c7210e489b67e81d64bc7aa3215ff940f36e4f85d95bf06b50" gracePeriod=45
Aug 01 00:46:56.939572 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:56.939510    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng"
Aug 01 00:46:56.975336 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:56.975303    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7694cb77-6106-4868-8ed8-58b2bb10eae2-tls-certs\") pod \"7694cb77-6106-4868-8ed8-58b2bb10eae2\" (UID: \"7694cb77-6106-4868-8ed8-58b2bb10eae2\") "
Aug 01 00:46:56.975519 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:56.975349    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xktnb\" (UniqueName: \"kubernetes.io/projected/7694cb77-6106-4868-8ed8-58b2bb10eae2-kube-api-access-xktnb\") pod \"7694cb77-6106-4868-8ed8-58b2bb10eae2\" (UID: \"7694cb77-6106-4868-8ed8-58b2bb10eae2\") "
Aug 01 00:46:56.977610 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:56.977576    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7694cb77-6106-4868-8ed8-58b2bb10eae2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7694cb77-6106-4868-8ed8-58b2bb10eae2" (UID: "7694cb77-6106-4868-8ed8-58b2bb10eae2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:46:56.977749 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:56.977655    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7694cb77-6106-4868-8ed8-58b2bb10eae2-kube-api-access-xktnb" (OuterVolumeSpecName: "kube-api-access-xktnb") pod "7694cb77-6106-4868-8ed8-58b2bb10eae2" (UID: "7694cb77-6106-4868-8ed8-58b2bb10eae2"). InnerVolumeSpecName "kube-api-access-xktnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:46:57.076162 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:57.076123    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7694cb77-6106-4868-8ed8-58b2bb10eae2-tls-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:46:57.076162 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:57.076156    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xktnb\" (UniqueName: \"kubernetes.io/projected/7694cb77-6106-4868-8ed8-58b2bb10eae2-kube-api-access-xktnb\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:46:57.670530 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:57.670494    2578 generic.go:356] "Generic (PLEG): container finished" podID="7694cb77-6106-4868-8ed8-58b2bb10eae2" containerID="742e0e69fe4dd8c7210e489b67e81d64bc7aa3215ff940f36e4f85d95bf06b50" exitCode=0
Aug 01 00:46:57.670744 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:57.670560    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng"
Aug 01 00:46:57.670744 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:57.670580    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng" event={"ID":"7694cb77-6106-4868-8ed8-58b2bb10eae2","Type":"ContainerDied","Data":"742e0e69fe4dd8c7210e489b67e81d64bc7aa3215ff940f36e4f85d95bf06b50"}
Aug 01 00:46:57.670744 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:57.670674    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng" event={"ID":"7694cb77-6106-4868-8ed8-58b2bb10eae2","Type":"ContainerDied","Data":"58ace1e14c3596ebaece24d94ed20f27a1dce40407079ad47624d827b381a527"}
Aug 01 00:46:57.670744 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:57.670704    2578 scope.go:117] "RemoveContainer" containerID="742e0e69fe4dd8c7210e489b67e81d64bc7aa3215ff940f36e4f85d95bf06b50"
Aug 01 00:46:57.680326 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:57.680309    2578 scope.go:117] "RemoveContainer" containerID="742e0e69fe4dd8c7210e489b67e81d64bc7aa3215ff940f36e4f85d95bf06b50"
Aug 01 00:46:57.680639 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:46:57.680619    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"742e0e69fe4dd8c7210e489b67e81d64bc7aa3215ff940f36e4f85d95bf06b50\": container with ID starting with 742e0e69fe4dd8c7210e489b67e81d64bc7aa3215ff940f36e4f85d95bf06b50 not found: ID does not exist" containerID="742e0e69fe4dd8c7210e489b67e81d64bc7aa3215ff940f36e4f85d95bf06b50"
Aug 01 00:46:57.680682 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:57.680650    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742e0e69fe4dd8c7210e489b67e81d64bc7aa3215ff940f36e4f85d95bf06b50"} err="failed to get container status \"742e0e69fe4dd8c7210e489b67e81d64bc7aa3215ff940f36e4f85d95bf06b50\": rpc error: code = NotFound desc = could not find container \"742e0e69fe4dd8c7210e489b67e81d64bc7aa3215ff940f36e4f85d95bf06b50\": container with ID starting with 742e0e69fe4dd8c7210e489b67e81d64bc7aa3215ff940f36e4f85d95bf06b50 not found: ID does not exist"
Aug 01 00:46:57.686118 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:57.686091    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng"]
Aug 01 00:46:57.689979 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:57.689954    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-9dqng"]
Aug 01 00:46:59.186575 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:46:59.186538    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7694cb77-6106-4868-8ed8-58b2bb10eae2" path="/var/lib/kubelet/pods/7694cb77-6106-4868-8ed8-58b2bb10eae2/volumes"
Aug 01 00:49:06.824149 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:06.824066    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m574z/must-gather-bmxv2"]
Aug 01 00:49:06.824568 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:06.824498    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7694cb77-6106-4868-8ed8-58b2bb10eae2" containerName="main"
Aug 01 00:49:06.824568 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:06.824512    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7694cb77-6106-4868-8ed8-58b2bb10eae2" containerName="main"
Aug 01 00:49:06.824568 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:06.824533    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b4d3515-8945-413d-a60b-fb089cd06e7d" containerName="main"
Aug 01 00:49:06.824568 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:06.824540    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b4d3515-8945-413d-a60b-fb089cd06e7d" containerName="main"
Aug 01 00:49:06.824705 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:06.824597    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b4d3515-8945-413d-a60b-fb089cd06e7d" containerName="main"
Aug 01 00:49:06.824705 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:06.824610    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7694cb77-6106-4868-8ed8-58b2bb10eae2" containerName="main"
Aug 01 00:49:06.827674 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:06.827657    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m574z/must-gather-bmxv2"
Aug 01 00:49:06.830403 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:06.830373    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m574z\"/\"kube-root-ca.crt\""
Aug 01 00:49:06.830617 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:06.830411    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m574z\"/\"openshift-service-ca.crt\""
Aug 01 00:49:06.830841 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:06.830820    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m574z\"/\"default-dockercfg-wxxnx\""
Aug 01 00:49:06.841927 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:06.841900    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m574z/must-gather-bmxv2"]
Aug 01 00:49:06.935618 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:06.935586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj2rw\" (UniqueName: \"kubernetes.io/projected/ef9fc9ef-773c-42b5-93b7-db1d63d3ad03-kube-api-access-fj2rw\") pod \"must-gather-bmxv2\" (UID: \"ef9fc9ef-773c-42b5-93b7-db1d63d3ad03\") " pod="openshift-must-gather-m574z/must-gather-bmxv2"
Aug 01 00:49:06.935822 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:06.935634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ef9fc9ef-773c-42b5-93b7-db1d63d3ad03-must-gather-output\") pod \"must-gather-bmxv2\" (UID: \"ef9fc9ef-773c-42b5-93b7-db1d63d3ad03\") " pod="openshift-must-gather-m574z/must-gather-bmxv2"
Aug 01 00:49:07.036774 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:07.036741    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj2rw\" (UniqueName: \"kubernetes.io/projected/ef9fc9ef-773c-42b5-93b7-db1d63d3ad03-kube-api-access-fj2rw\") pod \"must-gather-bmxv2\" (UID: \"ef9fc9ef-773c-42b5-93b7-db1d63d3ad03\") " pod="openshift-must-gather-m574z/must-gather-bmxv2"
Aug 01 00:49:07.037006 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:07.036784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ef9fc9ef-773c-42b5-93b7-db1d63d3ad03-must-gather-output\") pod \"must-gather-bmxv2\" (UID: \"ef9fc9ef-773c-42b5-93b7-db1d63d3ad03\") " pod="openshift-must-gather-m574z/must-gather-bmxv2"
Aug 01 00:49:07.037090 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:07.037075    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ef9fc9ef-773c-42b5-93b7-db1d63d3ad03-must-gather-output\") pod \"must-gather-bmxv2\" (UID: \"ef9fc9ef-773c-42b5-93b7-db1d63d3ad03\") " pod="openshift-must-gather-m574z/must-gather-bmxv2"
Aug 01 00:49:07.044944 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:07.044920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj2rw\" (UniqueName: \"kubernetes.io/projected/ef9fc9ef-773c-42b5-93b7-db1d63d3ad03-kube-api-access-fj2rw\") pod \"must-gather-bmxv2\" (UID: \"ef9fc9ef-773c-42b5-93b7-db1d63d3ad03\") " pod="openshift-must-gather-m574z/must-gather-bmxv2"
Aug 01 00:49:07.137787 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:07.137654    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m574z/must-gather-bmxv2"
Aug 01 00:49:07.276085 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:07.276057    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m574z/must-gather-bmxv2"]
Aug 01 00:49:07.278370 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:49:07.278345    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef9fc9ef_773c_42b5_93b7_db1d63d3ad03.slice/crio-ca87850a307d2530d3fc7f0eeb8b249de97299d70f95cd5a6841f0ad9b9dc827 WatchSource:0}: Error finding container ca87850a307d2530d3fc7f0eeb8b249de97299d70f95cd5a6841f0ad9b9dc827: Status 404 returned error can't find the container with id ca87850a307d2530d3fc7f0eeb8b249de97299d70f95cd5a6841f0ad9b9dc827
Aug 01 00:49:07.280459 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:07.280444    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:49:07.373290 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:07.373251    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m574z/must-gather-bmxv2" event={"ID":"ef9fc9ef-773c-42b5-93b7-db1d63d3ad03","Type":"ContainerStarted","Data":"ca87850a307d2530d3fc7f0eeb8b249de97299d70f95cd5a6841f0ad9b9dc827"}
Aug 01 00:49:14.422518 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:14.422477    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m574z/must-gather-bmxv2" event={"ID":"ef9fc9ef-773c-42b5-93b7-db1d63d3ad03","Type":"ContainerStarted","Data":"d8ff24b684173ec094e5b87535f78e9a04a36f55c7701a6b34cf00f78227e9f8"}
Aug 01 00:49:14.422987 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:14.422533    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m574z/must-gather-bmxv2" event={"ID":"ef9fc9ef-773c-42b5-93b7-db1d63d3ad03","Type":"ContainerStarted","Data":"065d67c92e2551712060cc4b908df8729251748821c7bdb2773d0450272cf997"}
Aug 01 00:49:14.439231 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:14.439173    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m574z/must-gather-bmxv2" podStartSLOduration=2.191497916 podStartE2EDuration="8.439151399s" totalImagesPullingTime="6.247653483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:49:14.436100454 +0000 UTC m=+3793.795991495" watchObservedRunningTime="2026-08-01 00:49:14.439151399 +0000 UTC m=+3793.799042441"
Aug 01 00:49:29.939877 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:29.939843    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8sb7w_92da091f-e08b-4bc5-865f-9b153946f58d/discovery/0.log"
Aug 01 00:49:30.915696 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:30.915610    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8sb7w_92da091f-e08b-4bc5-865f-9b153946f58d/discovery/0.log"
Aug 01 00:49:31.862306 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:31.862272    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8sb7w_92da091f-e08b-4bc5-865f-9b153946f58d/discovery/0.log"
Aug 01 00:49:32.832493 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:32.832457    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8sb7w_92da091f-e08b-4bc5-865f-9b153946f58d/discovery/0.log"
Aug 01 00:49:33.917145 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:33.917116    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:33.944507 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:33.944480    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:34.153521 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:34.153488    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:34.179411 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:34.179316    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:35.435990 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:35.435951    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:35.468938 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:35.468905    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:35.690751 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:35.690617    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:35.829128 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:35.829100    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:37.200959 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:37.200917    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:37.216258 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:37.216231    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:37.421528 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:37.421488    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:37.461019 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:37.460954    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:38.600973 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:38.600944    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:38.614630 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:38.614605    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:38.817326 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:38.817292    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:38.848343 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:38.848306    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:39.881547 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:39.881515    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:39.898168 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:39.898143    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:40.092175 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:40.092140    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:40.140033 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:40.139958    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:41.199030 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:41.199000    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:41.217200 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:41.217168    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:41.417654 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:41.417588    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:41.451843 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:41.451774    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:42.507424 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:42.507393    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:42.522175 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:42.522150    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:42.722768 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:42.722732    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:42.757378 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:42.757351    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:43.805466 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:43.805423    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:43.818261 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:43.818236    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:44.019698 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:44.019665    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:44.049990 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:44.049965    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:45.100342 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:45.100307    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:45.115671 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:45.115638    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:45.315946 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:45.315908    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:45.348247 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:45.348218    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:46.425437 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:46.425401    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:46.437778 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:46.437747    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:46.642240 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:46.642195    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:46.672615 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:46.672586    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:47.736995 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:47.736956    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:47.749764 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:47.749744    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:47.954456 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:47.954427    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:47.990772 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:47.990688    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:49.050399 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:49.050311    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:49.063940 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:49.063914    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:49.256656 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:49.256618    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:49.288267 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:49.288237    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:50.360359 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:50.360331    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:50.373759 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:50.373732    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:50.581472 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:50.581435    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:50.616223 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:50.616125    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:51.661365 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:51.661336    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:51.674825 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:51.674794    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:51.879905 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:51.879875    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:51.908071 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:51.908044    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:52.948124 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:52.948092    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:52.962497 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:52.962468    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:53.168391 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:53.168356    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:53.205875 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:53.205796    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:54.260306 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:54.260280    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:54.274762 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:54.274740    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:54.482379 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:54.482347    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:54.511210 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:54.511117    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:55.558820 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:55.558785    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:55.572675 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:55.572648    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:55.775228 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:55.775197    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:55.805671 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:55.805643    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:56.841903 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:56.841874    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:56.855303 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:56.855276    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:57.049975 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:57.049939    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:57.084145 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:57.084112    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:58.151237 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:58.151193    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:58.164603 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:58.164580    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:58.369319 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:58.369284    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:58.403003 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:58.402912    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:49:59.458021 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:59.457992    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:49:59.470991 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:59.470954    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:49:59.677600 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:59.677565    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:49:59.706397 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:49:59.706354    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:50:00.754233 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:00.754200    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:50:00.767269 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:00.767245    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:50:00.968675 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:00.968644    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:50:00.999540 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:00.999508    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:50:02.067360 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:02.067322    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:50:02.080906 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:02.080878    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:50:02.283771 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:02.283727    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:50:02.318651 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:02.318577    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:50:03.399879 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:03.399850    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:50:03.413950 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:03.413920    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:50:03.623606 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:03.623572    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:50:03.652679 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:03.652600    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:50:04.734729 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:04.734686    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:50:04.751742 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:04.751690    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:50:04.969906 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:04.969869    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:50:04.993654 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:04.993580    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:50:06.056097 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:06.056068    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:50:06.069708 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:06.069678    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:50:06.281506 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:06.281467    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:50:06.315170 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:06.315094    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:50:07.753018 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:07.752985    2578 generic.go:356] "Generic (PLEG): container finished" podID="ef9fc9ef-773c-42b5-93b7-db1d63d3ad03" containerID="065d67c92e2551712060cc4b908df8729251748821c7bdb2773d0450272cf997" exitCode=0
Aug 01 00:50:07.753523 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:07.753068    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m574z/must-gather-bmxv2" event={"ID":"ef9fc9ef-773c-42b5-93b7-db1d63d3ad03","Type":"ContainerDied","Data":"065d67c92e2551712060cc4b908df8729251748821c7bdb2773d0450272cf997"}
Aug 01 00:50:07.753523 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:07.753437    2578 scope.go:117] "RemoveContainer" containerID="065d67c92e2551712060cc4b908df8729251748821c7bdb2773d0450272cf997"
Aug 01 00:50:07.828608 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:07.828579    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-m574z_must-gather-bmxv2_ef9fc9ef-773c-42b5-93b7-db1d63d3ad03/gather/0.log"
Aug 01 00:50:11.189771 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:11.189741    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zzsl6_ab5415d7-3b45-4861-bfaf-be67c4d16871/global-pull-secret-syncer/0.log"
Aug 01 00:50:11.263015 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:11.262978    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7ng6n_1ddbfa22-7b8f-4800-8dd1-78a871bbf08d/konnectivity-agent/0.log"
Aug 01 00:50:11.377364 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:11.377338    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-3.ec2.internal_8a89e13fce40802bf99f22858790a870/haproxy/0.log"
Aug 01 00:50:13.273910 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.273870    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-m574z/must-gather-bmxv2"]
Aug 01 00:50:13.274405 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.274124    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-m574z/must-gather-bmxv2" podUID="ef9fc9ef-773c-42b5-93b7-db1d63d3ad03" containerName="copy" containerID="cri-o://d8ff24b684173ec094e5b87535f78e9a04a36f55c7701a6b34cf00f78227e9f8" gracePeriod=2
Aug 01 00:50:13.275860 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:50:13.275821    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bmxv2\" is forbidden: User \"system:node:ip-10-0-140-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m574z\": no relationship found between node 'ip-10-0-140-3.ec2.internal' and this object" podUID="ef9fc9ef-773c-42b5-93b7-db1d63d3ad03" pod="openshift-must-gather-m574z/must-gather-bmxv2"
Aug 01 00:50:13.277926 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.277895    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-m574z/must-gather-bmxv2"]
Aug 01 00:50:13.516691 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.516665    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-m574z_must-gather-bmxv2_ef9fc9ef-773c-42b5-93b7-db1d63d3ad03/copy/0.log"
Aug 01 00:50:13.519395 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.519374    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m574z/must-gather-bmxv2"
Aug 01 00:50:13.521092 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:50:13.521068    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bmxv2\" is forbidden: User \"system:node:ip-10-0-140-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m574z\": no relationship found between node 'ip-10-0-140-3.ec2.internal' and this object" podUID="ef9fc9ef-773c-42b5-93b7-db1d63d3ad03" pod="openshift-must-gather-m574z/must-gather-bmxv2"
Aug 01 00:50:13.628414 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.628332    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj2rw\" (UniqueName: \"kubernetes.io/projected/ef9fc9ef-773c-42b5-93b7-db1d63d3ad03-kube-api-access-fj2rw\") pod \"ef9fc9ef-773c-42b5-93b7-db1d63d3ad03\" (UID: \"ef9fc9ef-773c-42b5-93b7-db1d63d3ad03\") "
Aug 01 00:50:13.628414 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.628373    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ef9fc9ef-773c-42b5-93b7-db1d63d3ad03-must-gather-output\") pod \"ef9fc9ef-773c-42b5-93b7-db1d63d3ad03\" (UID: \"ef9fc9ef-773c-42b5-93b7-db1d63d3ad03\") "
Aug 01 00:50:13.630700 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.630663    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9fc9ef-773c-42b5-93b7-db1d63d3ad03-kube-api-access-fj2rw" (OuterVolumeSpecName: "kube-api-access-fj2rw") pod "ef9fc9ef-773c-42b5-93b7-db1d63d3ad03" (UID: "ef9fc9ef-773c-42b5-93b7-db1d63d3ad03"). InnerVolumeSpecName "kube-api-access-fj2rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:50:13.635017 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.634989    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9fc9ef-773c-42b5-93b7-db1d63d3ad03-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "ef9fc9ef-773c-42b5-93b7-db1d63d3ad03" (UID: "ef9fc9ef-773c-42b5-93b7-db1d63d3ad03"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:50:13.729424 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.729380    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj2rw\" (UniqueName: \"kubernetes.io/projected/ef9fc9ef-773c-42b5-93b7-db1d63d3ad03-kube-api-access-fj2rw\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:50:13.729424 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.729415    2578 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ef9fc9ef-773c-42b5-93b7-db1d63d3ad03-must-gather-output\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\""
Aug 01 00:50:13.787182 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.787156    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-m574z_must-gather-bmxv2_ef9fc9ef-773c-42b5-93b7-db1d63d3ad03/copy/0.log"
Aug 01 00:50:13.787515 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.787497    2578 generic.go:356] "Generic (PLEG): container finished" podID="ef9fc9ef-773c-42b5-93b7-db1d63d3ad03" containerID="d8ff24b684173ec094e5b87535f78e9a04a36f55c7701a6b34cf00f78227e9f8" exitCode=143
Aug 01 00:50:13.787577 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.787540    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m574z/must-gather-bmxv2"
Aug 01 00:50:13.787641 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.787609    2578 scope.go:117] "RemoveContainer" containerID="d8ff24b684173ec094e5b87535f78e9a04a36f55c7701a6b34cf00f78227e9f8"
Aug 01 00:50:13.789221 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:50:13.789195    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bmxv2\" is forbidden: User \"system:node:ip-10-0-140-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m574z\": no relationship found between node 'ip-10-0-140-3.ec2.internal' and this object" podUID="ef9fc9ef-773c-42b5-93b7-db1d63d3ad03" pod="openshift-must-gather-m574z/must-gather-bmxv2"
Aug 01 00:50:13.796644 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.796628    2578 scope.go:117] "RemoveContainer" containerID="065d67c92e2551712060cc4b908df8729251748821c7bdb2773d0450272cf997"
Aug 01 00:50:13.797078 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:50:13.797053    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bmxv2\" is forbidden: User \"system:node:ip-10-0-140-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m574z\": no relationship found between node 'ip-10-0-140-3.ec2.internal' and this object" podUID="ef9fc9ef-773c-42b5-93b7-db1d63d3ad03" pod="openshift-must-gather-m574z/must-gather-bmxv2"
Aug 01 00:50:13.809212 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.809196    2578 scope.go:117] "RemoveContainer" containerID="d8ff24b684173ec094e5b87535f78e9a04a36f55c7701a6b34cf00f78227e9f8"
Aug 01 00:50:13.809512 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:50:13.809494    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ff24b684173ec094e5b87535f78e9a04a36f55c7701a6b34cf00f78227e9f8\": container with ID starting with d8ff24b684173ec094e5b87535f78e9a04a36f55c7701a6b34cf00f78227e9f8 not found: ID does not exist" containerID="d8ff24b684173ec094e5b87535f78e9a04a36f55c7701a6b34cf00f78227e9f8"
Aug 01 00:50:13.809593 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.809519    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ff24b684173ec094e5b87535f78e9a04a36f55c7701a6b34cf00f78227e9f8"} err="failed to get container status \"d8ff24b684173ec094e5b87535f78e9a04a36f55c7701a6b34cf00f78227e9f8\": rpc error: code = NotFound desc = could not find container \"d8ff24b684173ec094e5b87535f78e9a04a36f55c7701a6b34cf00f78227e9f8\": container with ID starting with d8ff24b684173ec094e5b87535f78e9a04a36f55c7701a6b34cf00f78227e9f8 not found: ID does not exist"
Aug 01 00:50:13.809593 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.809536    2578 scope.go:117] "RemoveContainer" containerID="065d67c92e2551712060cc4b908df8729251748821c7bdb2773d0450272cf997"
Aug 01 00:50:13.809845 ip-10-0-140-3 kubenswrapper[2578]: E0801 00:50:13.809827    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"065d67c92e2551712060cc4b908df8729251748821c7bdb2773d0450272cf997\": container with ID starting with 065d67c92e2551712060cc4b908df8729251748821c7bdb2773d0450272cf997 not found: ID does not exist" containerID="065d67c92e2551712060cc4b908df8729251748821c7bdb2773d0450272cf997"
Aug 01 00:50:13.809898 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:13.809849    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"065d67c92e2551712060cc4b908df8729251748821c7bdb2773d0450272cf997"} err="failed to get container status \"065d67c92e2551712060cc4b908df8729251748821c7bdb2773d0450272cf997\": rpc error: code = NotFound desc = could not find container \"065d67c92e2551712060cc4b908df8729251748821c7bdb2773d0450272cf997\": container with ID starting with 065d67c92e2551712060cc4b908df8729251748821c7bdb2773d0450272cf997 not found: ID does not exist"
Aug 01 00:50:15.185259 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:15.185220    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef9fc9ef-773c-42b5-93b7-db1d63d3ad03" path="/var/lib/kubelet/pods/ef9fc9ef-773c-42b5-93b7-db1d63d3ad03/volumes"
Aug 01 00:50:16.908517 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:16.908488    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-9ds2s_01d872a4-52b1-48d5-ab18-45fbd55718cd/manager/0.log"
Aug 01 00:50:16.945290 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:16.945261    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9kq7d_1a1ba4de-4d45-4d0c-b69b-1fbfa3adee26/manager/0.log"
Aug 01 00:50:17.146484 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:17.146449    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-7qwtd_33532000-c2e7-4881-ac4b-d27d581dbdad/manager/0.log"
Aug 01 00:50:17.221866 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:17.221793    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-2w49s_96ef8b85-f389-4d7f-916b-cc17bb35cf2f/manager/0.log"
Aug 01 00:50:18.254819 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:18.254789    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_184415c8-ee5b-4373-b59d-993997326934/alertmanager/0.log"
Aug 01 00:50:18.273472 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:18.273445    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_184415c8-ee5b-4373-b59d-993997326934/config-reloader/0.log"
Aug 01 00:50:18.291863 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:18.291843    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_184415c8-ee5b-4373-b59d-993997326934/kube-rbac-proxy-web/0.log"
Aug 01 00:50:18.309404 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:18.309387    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_184415c8-ee5b-4373-b59d-993997326934/kube-rbac-proxy/0.log"
Aug 01 00:50:18.327377 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:18.327352    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_184415c8-ee5b-4373-b59d-993997326934/kube-rbac-proxy-metric/0.log"
Aug 01 00:50:18.345866 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:18.345843    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_184415c8-ee5b-4373-b59d-993997326934/prom-label-proxy/0.log"
Aug 01 00:50:18.364448 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:18.364431    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_184415c8-ee5b-4373-b59d-993997326934/init-config-reloader/0.log"
Aug 01 00:50:18.440636 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:18.440605    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-r4f5l_46bbcd5c-1c67-4966-8290-90106568f72b/kube-state-metrics/0.log"
Aug 01 00:50:18.460991 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:18.460972    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-r4f5l_46bbcd5c-1c67-4966-8290-90106568f72b/kube-rbac-proxy-main/0.log"
Aug 01 00:50:18.479197 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:18.479179    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-r4f5l_46bbcd5c-1c67-4966-8290-90106568f72b/kube-rbac-proxy-self/0.log"
Aug 01 00:50:18.534083 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:18.534013    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-l7z68_311b3c95-8d5d-485a-8037-d473da6787d4/monitoring-plugin/0.log"
Aug 01 00:50:18.654228 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:18.654197    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-75gvm_2e5bad06-e384-45d4-8863-7d2563e0d648/node-exporter/0.log"
Aug 01 00:50:18.676748 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:18.676730    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-75gvm_2e5bad06-e384-45d4-8863-7d2563e0d648/kube-rbac-proxy/0.log"
Aug 01 00:50:18.713693 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:18.713670    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-75gvm_2e5bad06-e384-45d4-8863-7d2563e0d648/init-textfile/0.log"
Aug 01 00:50:18.836334 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:18.836258    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-lkdp8_6cbde346-ebc0-415d-85e7-c61319db6262/kube-rbac-proxy-main/0.log"
Aug 01 00:50:18.864020 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:18.863996    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-lkdp8_6cbde346-ebc0-415d-85e7-c61319db6262/kube-rbac-proxy-self/0.log"
Aug 01 00:50:18.885578 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:18.885553    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-lkdp8_6cbde346-ebc0-415d-85e7-c61319db6262/openshift-state-metrics/0.log"
Aug 01 00:50:19.077711 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:19.077676    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-dlctp_7a162687-ed57-4042-b0f8-8a41623fa94c/prometheus-operator/0.log"
Aug 01 00:50:19.119385 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:19.119315    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-dlctp_7a162687-ed57-4042-b0f8-8a41623fa94c/kube-rbac-proxy/0.log"
Aug 01 00:50:19.193998 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:19.193968    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b6dd4b7c9-g59jt_070d2df4-7d6c-42ae-ae2f-696260f6162a/telemeter-client/0.log"
Aug 01 00:50:19.214633 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:19.214600    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b6dd4b7c9-g59jt_070d2df4-7d6c-42ae-ae2f-696260f6162a/reload/0.log"
Aug 01 00:50:19.238735 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:19.238695    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b6dd4b7c9-g59jt_070d2df4-7d6c-42ae-ae2f-696260f6162a/kube-rbac-proxy/0.log"
Aug 01 00:50:21.285071 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.285030    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"]
Aug 01 00:50:21.285794 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.285689    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9fc9ef-773c-42b5-93b7-db1d63d3ad03" containerName="copy"
Aug 01 00:50:21.285794 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.285728    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9fc9ef-773c-42b5-93b7-db1d63d3ad03" containerName="copy"
Aug 01 00:50:21.285794 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.285753    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9fc9ef-773c-42b5-93b7-db1d63d3ad03" containerName="gather"
Aug 01 00:50:21.285794 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.285762    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9fc9ef-773c-42b5-93b7-db1d63d3ad03" containerName="gather"
Aug 01 00:50:21.286040 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.285855    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef9fc9ef-773c-42b5-93b7-db1d63d3ad03" containerName="copy"
Aug 01 00:50:21.286040 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.285871    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef9fc9ef-773c-42b5-93b7-db1d63d3ad03" containerName="gather"
Aug 01 00:50:21.287035 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.287014    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c56c5646b-pkdsf_361ef423-1058-4296-81de-f7df57bee743/console/0.log"
Aug 01 00:50:21.289164 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.289145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:21.290818 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.290801    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kbl7c\"/\"default-dockercfg-txszj\""
Aug 01 00:50:21.291028 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.291011    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kbl7c\"/\"openshift-service-ca.crt\""
Aug 01 00:50:21.291314 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.291298    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kbl7c\"/\"kube-root-ca.crt\""
Aug 01 00:50:21.298111 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.298082    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"]
Aug 01 00:50:21.393779 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.393736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a820e5e-9cd6-467d-8965-dfbca6bdac08-lib-modules\") pod \"perf-node-gather-daemonset-m4qwk\" (UID: \"8a820e5e-9cd6-467d-8965-dfbca6bdac08\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:21.393950 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.393797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a820e5e-9cd6-467d-8965-dfbca6bdac08-proc\") pod \"perf-node-gather-daemonset-m4qwk\" (UID: \"8a820e5e-9cd6-467d-8965-dfbca6bdac08\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:21.393950 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.393827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54hmj\" (UniqueName: \"kubernetes.io/projected/8a820e5e-9cd6-467d-8965-dfbca6bdac08-kube-api-access-54hmj\") pod \"perf-node-gather-daemonset-m4qwk\" (UID: \"8a820e5e-9cd6-467d-8965-dfbca6bdac08\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:21.393950 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.393853    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a820e5e-9cd6-467d-8965-dfbca6bdac08-sys\") pod \"perf-node-gather-daemonset-m4qwk\" (UID: \"8a820e5e-9cd6-467d-8965-dfbca6bdac08\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:21.393950 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.393881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a820e5e-9cd6-467d-8965-dfbca6bdac08-podres\") pod \"perf-node-gather-daemonset-m4qwk\" (UID: \"8a820e5e-9cd6-467d-8965-dfbca6bdac08\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:21.495090 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.495052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54hmj\" (UniqueName: \"kubernetes.io/projected/8a820e5e-9cd6-467d-8965-dfbca6bdac08-kube-api-access-54hmj\") pod \"perf-node-gather-daemonset-m4qwk\" (UID: \"8a820e5e-9cd6-467d-8965-dfbca6bdac08\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:21.495286 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.495116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a820e5e-9cd6-467d-8965-dfbca6bdac08-sys\") pod \"perf-node-gather-daemonset-m4qwk\" (UID: \"8a820e5e-9cd6-467d-8965-dfbca6bdac08\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:21.495286 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.495160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a820e5e-9cd6-467d-8965-dfbca6bdac08-podres\") pod \"perf-node-gather-daemonset-m4qwk\" (UID: \"8a820e5e-9cd6-467d-8965-dfbca6bdac08\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:21.495286 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.495232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a820e5e-9cd6-467d-8965-dfbca6bdac08-lib-modules\") pod \"perf-node-gather-daemonset-m4qwk\" (UID: \"8a820e5e-9cd6-467d-8965-dfbca6bdac08\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:21.495286 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.495251    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a820e5e-9cd6-467d-8965-dfbca6bdac08-sys\") pod \"perf-node-gather-daemonset-m4qwk\" (UID: \"8a820e5e-9cd6-467d-8965-dfbca6bdac08\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:21.495486 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.495289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a820e5e-9cd6-467d-8965-dfbca6bdac08-proc\") pod \"perf-node-gather-daemonset-m4qwk\" (UID: \"8a820e5e-9cd6-467d-8965-dfbca6bdac08\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:21.495486 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.495370    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a820e5e-9cd6-467d-8965-dfbca6bdac08-podres\") pod \"perf-node-gather-daemonset-m4qwk\" (UID: \"8a820e5e-9cd6-467d-8965-dfbca6bdac08\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:21.495486 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.495387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a820e5e-9cd6-467d-8965-dfbca6bdac08-proc\") pod \"perf-node-gather-daemonset-m4qwk\" (UID: \"8a820e5e-9cd6-467d-8965-dfbca6bdac08\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:21.495486 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.495409    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a820e5e-9cd6-467d-8965-dfbca6bdac08-lib-modules\") pod \"perf-node-gather-daemonset-m4qwk\" (UID: \"8a820e5e-9cd6-467d-8965-dfbca6bdac08\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:21.503187 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.503168    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54hmj\" (UniqueName: \"kubernetes.io/projected/8a820e5e-9cd6-467d-8965-dfbca6bdac08-kube-api-access-54hmj\") pod \"perf-node-gather-daemonset-m4qwk\" (UID: \"8a820e5e-9cd6-467d-8965-dfbca6bdac08\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:21.600574 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.600476    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:21.740349 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.740319    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"]
Aug 01 00:50:21.742458 ip-10-0-140-3 kubenswrapper[2578]: W0801 00:50:21.742433    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8a820e5e_9cd6_467d_8965_dfbca6bdac08.slice/crio-54037722f9f9b874aacc73703ab10cf1219f8517751b4835ddc901f0e5de25e6 WatchSource:0}: Error finding container 54037722f9f9b874aacc73703ab10cf1219f8517751b4835ddc901f0e5de25e6: Status 404 returned error can't find the container with id 54037722f9f9b874aacc73703ab10cf1219f8517751b4835ddc901f0e5de25e6
Aug 01 00:50:21.841390 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.841344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk" event={"ID":"8a820e5e-9cd6-467d-8965-dfbca6bdac08","Type":"ContainerStarted","Data":"402f1de451c70dd414afac80fdd471253347382fcdd929d6bf2bf206e2bf85fc"}
Aug 01 00:50:21.841552 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.841429    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk" event={"ID":"8a820e5e-9cd6-467d-8965-dfbca6bdac08","Type":"ContainerStarted","Data":"54037722f9f9b874aacc73703ab10cf1219f8517751b4835ddc901f0e5de25e6"}
Aug 01 00:50:21.841552 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.841452    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:21.855840 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:21.855747    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk" podStartSLOduration=0.855710169 podStartE2EDuration="855.710169ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:50:21.854847057 +0000 UTC m=+3861.214738122" watchObservedRunningTime="2026-08-01 00:50:21.855710169 +0000 UTC m=+3861.215601213"
Aug 01 00:50:22.845810 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:22.845782    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5nwm2_1de6b043-026d-4c14-a302-25c6357dc375/dns/0.log"
Aug 01 00:50:22.864429 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:22.864404    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5nwm2_1de6b043-026d-4c14-a302-25c6357dc375/kube-rbac-proxy/0.log"
Aug 01 00:50:22.976675 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:22.976648    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5r99m_d1a3dcea-7f41-400b-b402-9cf780a6b030/dns-node-resolver/0.log"
Aug 01 00:50:23.580542 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:23.580516    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jxfkn_c459a9c2-d75c-44d9-865b-4262edeb4f61/node-ca/0.log"
Aug 01 00:50:24.655759 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:24.655709    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8sb7w_92da091f-e08b-4bc5-865f-9b153946f58d/discovery/0.log"
Aug 01 00:50:25.280432 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:25.280397    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gvwvq_93a82855-64d9-4eb2-bdc5-fb4cfef63f88/serve-healthcheck-canary/0.log"
Aug 01 00:50:25.834038 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:25.834011    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5qwfc_7f1719f9-a204-4ba5-bf61-1197626af667/kube-rbac-proxy/0.log"
Aug 01 00:50:25.881732 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:25.881685    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5qwfc_7f1719f9-a204-4ba5-bf61-1197626af667/exporter/0.log"
Aug 01 00:50:25.931327 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:25.931304    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5qwfc_7f1719f9-a204-4ba5-bf61-1197626af667/extractor/0.log"
Aug 01 00:50:27.857402 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:27.857372    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-m4qwk"
Aug 01 00:50:30.079475 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:30.079449    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-674d959fb8-zbm2g_5dbb392c-4aab-4c3e-ac0e-7f578ff3a398/manager/0.log"
Aug 01 00:50:30.503116 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:30.503089    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-95fqk_72dc53a4-36dc-4542-9571-6c7ecd502cae/server/0.log"
Aug 01 00:50:30.507339 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:30.507316    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8sb7w_92da091f-e08b-4bc5-865f-9b153946f58d/discovery/0.log"
Aug 01 00:50:31.336550 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:31.336516    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-8588677cfb-srd4w_c6aed191-891b-4fb8-ae0c-96e786b71747/manager/0.log"
Aug 01 00:50:31.429296 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:31.429263    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-tgv9k_01882f1e-fae3-4c5c-a021-024b971ede1a/seaweedfs/0.log"
Aug 01 00:50:31.928649 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:31.928611    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8sb7w_92da091f-e08b-4bc5-865f-9b153946f58d/discovery/0.log"
Aug 01 00:50:37.191021 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:37.190989    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6jv4r_7405cb77-3fa3-47ae-864d-b8b59f586932/kube-multus-additional-cni-plugins/0.log"
Aug 01 00:50:37.210363 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:37.210342    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6jv4r_7405cb77-3fa3-47ae-864d-b8b59f586932/egress-router-binary-copy/0.log"
Aug 01 00:50:37.229076 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:37.229049    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6jv4r_7405cb77-3fa3-47ae-864d-b8b59f586932/cni-plugins/0.log"
Aug 01 00:50:37.250292 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:37.250272    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6jv4r_7405cb77-3fa3-47ae-864d-b8b59f586932/bond-cni-plugin/0.log"
Aug 01 00:50:37.294545 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:37.294515    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6jv4r_7405cb77-3fa3-47ae-864d-b8b59f586932/routeoverride-cni/0.log"
Aug 01 00:50:37.313450 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:37.313424    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6jv4r_7405cb77-3fa3-47ae-864d-b8b59f586932/whereabouts-cni-bincopy/0.log"
Aug 01 00:50:37.335169 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:37.335140    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6jv4r_7405cb77-3fa3-47ae-864d-b8b59f586932/whereabouts-cni/0.log"
Aug 01 00:50:37.560925 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:37.560842    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ppq4j_345890db-ca81-4c74-a721-ba0904c50bf4/kube-multus/0.log"
Aug 01 00:50:37.621258 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:37.621233    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-js2f5_ec95cdf0-c411-409a-a0e5-274712ef6c00/network-metrics-daemon/0.log"
Aug 01 00:50:37.639336 ip-10-0-140-3 kubenswrapper[2578]: I0801 00:50:37.639317    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-js2f5_ec95cdf0-c411-409a-a0e5-274712ef6c00/kube-rbac-proxy/0.log"