Jul 20 09:56:26.119730 ip-10-0-133-160 systemd[1]: Starting Kubernetes Kubelet...
Jul 20 09:56:26.500069 ip-10-0-133-160 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 09:56:26.500069 ip-10-0-133-160 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 20 09:56:26.500069 ip-10-0-133-160 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 09:56:26.500069 ip-10-0-133-160 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 20 09:56:26.500069 ip-10-0-133-160 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 09:56:26.500818 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.500732    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 20 09:56:26.505383 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505368    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:26.505383 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505383    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505388    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505391    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505394    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505397    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505401    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505404    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505407    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505410    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505413    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505416    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505418    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505439    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505443    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505445    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505448    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505450    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505453    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505456    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505458    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:26.505465 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505461    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505463    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505466    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505468    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505471    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505474    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505477    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505479    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505482    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505485    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505488    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505490    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505492    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505495    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505497    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505500    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505502    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505505    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505507    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505509    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:26.505939 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505512    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505514    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505517    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505519    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505522    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505525    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505527    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505530    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505532    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505534    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505537    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505539    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505541    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505544    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505547    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505550    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505552    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505555    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505557    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505560    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:26.506445 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505563    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505565    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505568    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505570    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505575    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505578    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505581    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505584    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505586    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505589    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505592    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505594    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505597    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505599    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505602    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505604    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505606    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505609    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505613    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:26.507195 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505615    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505618    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505620    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505623    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505625    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505628    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505630    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505633    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505637    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505640    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505642    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505645    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505648    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505652    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505655    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505658    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505661    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505664    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.505667    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:26.507699 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507656    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507669    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507672    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507675    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507678    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507681    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507684    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507687    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507690    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507692    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507695    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507698    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507701    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507704    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507706    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507711    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507713    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507725    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507732    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:26.508213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507736    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507739    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507742    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507744    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507748    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507751    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507754    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507757    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507759    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507762    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507764    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507767    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507769    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507778    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507781    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507784    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507787    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507789    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507792    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:26.508710 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507797    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507801    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507806    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507808    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507812    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507814    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507817    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507820    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507822    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507825    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507827    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507830    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507832    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507835    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507837    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507840    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507842    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507845    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507847    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507850    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:26.509170 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507853    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507856    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507858    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507861    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507863    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507865    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507868    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507870    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507873    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507876    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507878    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507881    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507883    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507886    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507889    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507892    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507894    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507896    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507899    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507901    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:26.509674 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507904    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507906    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507908    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507911    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507913    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507917    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507920    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507923    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507925    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507927    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507930    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507932    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507935    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507937    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507940    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507942    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507945    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507947    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507950    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507952    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:26.510159 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.507955    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508339    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508344    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508346    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508349    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508351    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508354    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508357    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508360    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508362    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508365    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508369    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508373    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508376    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508379    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508382    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508385    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508388    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508391    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508394    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:26.510667 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508397    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508399    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508402    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508405    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508407    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508410    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508412    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508415    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508417    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508420    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508422    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508424    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508450    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508453    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508456    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508458    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508461    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508464    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508467    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:26.511143 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508469    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508473    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508476    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508479    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508481    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508484    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508486    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508489    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508492    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508494    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508497    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508503    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508505    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508508    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508510    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508513    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508516    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508518    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508521    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508523    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:26.511620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508526    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508528    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508531    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508536    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508539    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508542    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508545    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508548    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508550    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508553    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508555    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508558    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508560    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508562    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508565    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508567    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508570    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508572    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508575    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:26.512104 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508577    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508580    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508582    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508584    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508588    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508595    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508598    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508600    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508603    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508605    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508607    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508610    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508613    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508615    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508618    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508620    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508622    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508625    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508627    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508630    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:26.512620 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508632    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.508634    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508700    2580 flags.go:64] FLAG: --address="0.0.0.0"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508707    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508713    2580 flags.go:64] FLAG: --anonymous-auth="true"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508717    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508721    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508725    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508729    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508733    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508736    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508739    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508742    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508745    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508748    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508752    2580 flags.go:64] FLAG: --cgroup-root=""
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508756    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508759    2580 flags.go:64] FLAG: --client-ca-file=""
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508762    2580 flags.go:64] FLAG: --cloud-provider="external"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508765    2580 flags.go:64] FLAG: --cluster-dns="[]"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508769    2580 flags.go:64] FLAG: --cluster-domain=""
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508772    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508775    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508777    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 20 09:56:26.513178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508781    2580 flags.go:64] FLAG: --container-log-max-files="5"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508784    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508787    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508790    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508793    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508796    2580 flags.go:64] FLAG: --contention-profiling="false"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508799    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508801    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508804    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508807    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508811    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508814    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508817    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508820    2580 flags.go:64] FLAG: --enable-load-reader="false"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508823    2580 flags.go:64] FLAG: --enable-server="true"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508826    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508830    2580 flags.go:64] FLAG: --event-burst="100"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508833    2580 flags.go:64] FLAG: --event-qps="50"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508836    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508838    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508841    2580 flags.go:64] FLAG: --eviction-hard=""
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508849    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508852    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508855    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508858    2580 flags.go:64] FLAG: --eviction-soft=""
Jul 20 09:56:26.513793 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508861    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508864    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508868    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508870    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508873    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508876    2580 flags.go:64] FLAG: --fail-swap-on="true"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508879    2580 flags.go:64] FLAG: --feature-gates=""
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508882    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508885    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508888    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508891    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508893    2580 flags.go:64] FLAG: --healthz-port="10248"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508896    2580 flags.go:64] FLAG: --help="false"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508899    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-133-160.ec2.internal"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508901    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508904    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508907    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508910    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508914    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508917    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508919    2580 flags.go:64] FLAG: --image-service-endpoint=""
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508922    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508925    2580 flags.go:64] FLAG: --kube-api-burst="100"
Jul 20 09:56:26.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508928    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508931    2580 flags.go:64] FLAG: --kube-api-qps="50"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508934    2580 flags.go:64] FLAG: --kube-reserved=""
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508936    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508939    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508942    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508944    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508947    2580 flags.go:64] FLAG: --lock-file=""
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508950    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508953    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508956    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508961    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508965    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508972    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508975    2580 flags.go:64] FLAG: --logging-format="text"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508978    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508981    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508984    2580 flags.go:64] FLAG: --manifest-url=""
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508986    2580 flags.go:64] FLAG: --manifest-url-header=""
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508990    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508993    2580 flags.go:64] FLAG: --max-open-files="1000000"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.508997    2580 flags.go:64] FLAG: --max-pods="110"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509000    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509003    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509005    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 20 09:56:26.514978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509009    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509012    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509015    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509017    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509024    2580 flags.go:64] FLAG: --node-status-max-images="50"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509027    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509030    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509033    2580 flags.go:64] FLAG: --pod-cidr=""
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509035    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509041    2580 flags.go:64] FLAG: --pod-manifest-path=""
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509044    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509047    2580 flags.go:64] FLAG: --pods-per-core="0"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509049    2580 flags.go:64] FLAG: --port="10250"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509052    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509055    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0867aae1b1c621109"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509058    2580 flags.go:64] FLAG: --qos-reserved=""
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509061    2580 flags.go:64] FLAG: --read-only-port="10255"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509064    2580 flags.go:64] FLAG: --register-node="true"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509066    2580 flags.go:64] FLAG: --register-with-taints=""
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509070    2580 flags.go:64] FLAG: --registry-burst="10"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509073    2580 flags.go:64] FLAG: --registry-qps="5"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509075    2580 flags.go:64] FLAG: --reserved-cpus=""
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509078    2580 flags.go:64] FLAG: --reserved-memory=""
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509081    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509084    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 20 09:56:26.515620 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509087    2580 flags.go:64] FLAG: --rotate-certificates="false"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509089    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509092    2580 flags.go:64] FLAG: --runonce="false"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509095    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509098    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509100    2580 flags.go:64] FLAG: --seccomp-default="false"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509104    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509107    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509110    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509112    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509115    2580 flags.go:64] FLAG: --storage-driver-password="root"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509118    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509120    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509123    2580 flags.go:64] FLAG: --storage-driver-user="root"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509125    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509128    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509131    2580 flags.go:64] FLAG: --system-cgroups=""
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509133    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509138    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509141    2580 flags.go:64] FLAG: --tls-cert-file=""
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509143    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509147    2580 flags.go:64] FLAG: --tls-min-version=""
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509150    2580 flags.go:64] FLAG: --tls-private-key-file=""
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509152    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509155    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 20 09:56:26.516287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509158    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509160    2580 flags.go:64] FLAG: --v="2"
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509164    2580 flags.go:64] FLAG: --version="false"
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509169    2580 flags.go:64] FLAG: --vmodule=""
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509172    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.509175    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509292    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509296    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509299    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509302    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509305    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509307    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509310    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509314    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509316    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509319    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509321    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509323    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509326    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509328    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509331    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:26.516923 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509333    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509337    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509340    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509343    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509346    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509349    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509351    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509354    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509358    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509361    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509363    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509366    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509368    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509371    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509373    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509376    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509378    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509381    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509383    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:26.517446 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509386    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509388    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509391    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509393    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509396    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509400    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509402    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509405    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509407    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509409    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509412    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509414    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509417    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509419    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509422    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509424    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509443    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509446    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509449    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509451    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:26.517907 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509454    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509457    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509459    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509462    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509464    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509467    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509470    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509472    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509474    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509478    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509481    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509484    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509486    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509489    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509491    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509493    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509496    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509500    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509502    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509504    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:26.518403 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509507    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509509    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509512    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509514    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509518    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509521    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509524    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509527    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509529    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509532    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509534    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509537    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509539    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509545    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509549    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509551    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509554    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509557    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509559    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:26.518900 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509561    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:26.519360 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509564    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:26.519360 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509566    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:26.519360 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509570    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:26.519360 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509572    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:26.519360 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.509575    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:26.519360 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.510215    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 09:56:26.519360 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.517087    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 20 09:56:26.519360 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.517104    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 20 09:56:26.519360 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.517696    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 20 09:56:26.519360 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.517704    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 20 09:56:26.519360 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517762    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:26.519360 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517767    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:26.519360 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517770    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:26.519360 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517773    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:26.519360 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517776    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517779    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517782    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517785    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517788    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517790    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517792    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517795    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517798    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517800    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517802    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517805    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517807    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517810    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517812    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517815    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517817    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517819    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517822    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517824    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:26.519756 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517827    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517829    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517832    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517835    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517837    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517841    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517844    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517846    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517849    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517851    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517854    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517858    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517862    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517865    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517868    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517871    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517874    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517876    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517879    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:26.520239 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517881    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517884    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517886    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517889    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517891    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517894    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517896    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517899    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517901    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517904    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517907    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517910    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517912    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517915    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517918    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517920    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517924    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517929    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517933    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:26.520714 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517936    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517939    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517942    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517945    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517948    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517950    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517953    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517956    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517958    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517961    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517963    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517966    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517968    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517971    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517973    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517976    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517978    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517981    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517983    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517986    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:26.521213 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517988    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:26.521716 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517991    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:26.521716 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517993    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:26.521716 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517996    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:26.521716 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.517998    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:26.521716 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518001    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:26.521716 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518003    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:26.521716 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518006    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:26.521716 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518008    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:26.521716 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518010    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:26.521716 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518014    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:26.521716 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518016    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:26.521716 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518019    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:26.521716 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518022    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:26.521716 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518024    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:26.521716 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518026    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:26.521716 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518029    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:26.522098 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.518034    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 09:56:26.522098 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518146    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:26.522098 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518150    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:26.522098 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518153    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:26.522098 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518155    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:26.522098 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518158    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:26.522098 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518161    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:26.522098 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518163    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:26.522098 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518166    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:26.522098 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518168    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:26.522098 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518172    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:26.522098 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518176    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:26.522098 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518179    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:26.522098 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518182    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518184    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518187    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518190    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518192    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518195    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518198    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518200    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518203    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518205    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518207    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518210    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518212    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518215    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518217    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518220    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518224    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518227    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518230    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518233    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:26.522507 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518235    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518238    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518240    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518243    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518245    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518248    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518250    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518252    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518255    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518257    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518259    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518261    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518264    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518266    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518269    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518271    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518274    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518276    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518279    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518281    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:26.522989 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518284    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518286    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518288    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518290    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518293    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518295    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518298    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518300    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518302    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518305    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518307    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518310    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518312    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518315    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518317    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518319    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518322    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518324    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518326    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518329    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:26.523490 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518331    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518333    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518336    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518338    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518341    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518343    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518346    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518348    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518351    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518353    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518356    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518358    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518360    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518363    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518365    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518367    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518370    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518372    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518375    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518377    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:26.523977 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518379    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:26.524504 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518382    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:26.524504 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518385    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:26.524504 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518387    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:26.524504 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518389    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:26.524504 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518392    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:26.524504 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:26.518394    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:26.524504 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.518399    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 09:56:26.524504 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.518523    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 20 09:56:26.524504 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.520976    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 20 09:56:26.524504 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.522101    2580 server.go:1019] "Starting client certificate rotation"
Jul 20 09:56:26.524504 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.522196    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 20 09:56:26.524504 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.522233    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 20 09:56:26.524797 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.524517    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 20 09:56:26.525687 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.525669    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 20 09:56:26.540732 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.540715    2580 log.go:25] "Validated CRI v1 runtime API"
Jul 20 09:56:26.543595 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.543574    2580 log.go:25] "Validated CRI v1 image API"
Jul 20 09:56:26.544514 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.544492    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 20 09:56:26.548995 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.548975    2580 fs.go:135] Filesystem UUIDs: map[32227be1-a172-4e97-a4a1-24fe110a62e4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d7478c79-50f5-4d59-b450-88033eff31ba:/dev/nvme0n1p4]
Jul 20 09:56:26.549072 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.548996    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 20 09:56:26.552364 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.552346    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 20 09:56:26.554211 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.554108    2580 manager.go:216] Machine: {Timestamp:2026-07-20 09:56:26.553600229 +0000 UTC m=+0.315200789 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100014 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27fc4229aba040f16af98f32037906 SystemUUID:ec27fc42-29ab-a040-f16a-f98f32037906 BootID:e2eaad54-883e-4d79-adff-d55f64b0e370 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e8:da:20:48:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e8:da:20:48:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:10:4f:de:62:73 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 20 09:56:26.554211 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.554203    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 20 09:56:26.554349 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.554302    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 20 09:56:26.555477 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.555452    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 20 09:56:26.555655 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.555480    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-160.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 20 09:56:26.556206 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.556194    2580 topology_manager.go:138] "Creating topology manager with none policy"
Jul 20 09:56:26.556258 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.556212    2580 container_manager_linux.go:309] "Creating device plugin manager"
Jul 20 09:56:26.556258 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.556223    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 20 09:56:26.557024 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.557012    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 20 09:56:26.558190 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.558178    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 20 09:56:26.561075 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.561061    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 20 09:56:26.561208 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.561195    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 20 09:56:26.561263 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.561234    2580 kubelet.go:486] "Attempting to sync node with API server"
Jul 20 09:56:26.561263 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.561249    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 20 09:56:26.561366 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.561268    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 20 09:56:26.561366 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.561283    2580 kubelet.go:398] "Adding apiserver pod source"
Jul 20 09:56:26.561366 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.561295    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 20 09:56:26.561718 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.561695    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 20 09:56:26.561718 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.561720    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 20 09:56:26.564342 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.564327    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 20 09:56:26.566061 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.566048    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 20 09:56:26.566108 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.566078    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 20 09:56:26.567110 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.567097    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 20 09:56:26.567143 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.567117    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 20 09:56:26.567143 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.567124    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 20 09:56:26.567143 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.567137    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 20 09:56:26.567229 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.567146    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 20 09:56:26.567229 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.567152    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 20 09:56:26.567229 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.567160    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 20 09:56:26.567229 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.567165    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 20 09:56:26.567229 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.567171    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 20 09:56:26.567229 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.567176    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 20 09:56:26.567229 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.567188    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 20 09:56:26.567229 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.567195    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 20 09:56:26.567229 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.567222    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 20 09:56:26.567229 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.567228    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 20 09:56:26.570554 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.570541    2580 server.go:1268] "Started kubelet"
Jul 20 09:56:26.570779 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.570677    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 20 09:56:26.571148 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.570880    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 20 09:56:26.571148 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.571069    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 20 09:56:26.571573 ip-10-0-133-160 systemd[1]: Started Kubernetes Kubelet.
Jul 20 09:56:26.572376 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.572357    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 20 09:56:26.573287 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.573256    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 20 09:56:26.573287 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.573271    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 20 09:56:26.573482 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.573459    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-160.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 20 09:56:26.574226 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.574209    2580 server.go:310] "Adding debug handlers to kubelet server"
Jul 20 09:56:26.584404 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.584383    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 20 09:56:26.584404 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.584396    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 20 09:56:26.584893 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.584786    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 20 09:56:26.586270 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.585091    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 20 09:56:26.586270 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.585472    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 20 09:56:26.586270 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.585489    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 20 09:56:26.586270 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.585756    2580 reconstruct.go:97] "Volume reconstruction finished"
Jul 20 09:56:26.586270 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.585765    2580 reconciler.go:29] "Reconciler: start to sync state"
Jul 20 09:56:26.586563 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.586393    2580 factory.go:55] Registering systemd factory
Jul 20 09:56:26.586563 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.586407    2580 factory.go:223] Registration of the systemd container factory successfully
Jul 20 09:56:26.586957 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.586916    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 20 09:56:26.587046 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.583025    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-160.ec2.internal.18c3f6efa984b5fe  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-160.ec2.internal,UID:ip-10-0-133-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-160.ec2.internal,},FirstTimestamp:2026-07-20 09:56:26.57051187 +0000 UTC m=+0.332112429,LastTimestamp:2026-07-20 09:56:26.57051187 +0000 UTC m=+0.332112429,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-160.ec2.internal,}"
Jul 20 09:56:26.587112 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.587051    2580 factory.go:153] Registering CRI-O factory
Jul 20 09:56:26.587112 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.587064    2580 factory.go:223] Registration of the crio container factory successfully
Jul 20 09:56:26.587208 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.587150    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 20 09:56:26.587479 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.587461    2580 factory.go:103] Registering Raw factory
Jul 20 09:56:26.587533 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.587498    2580 manager.go:1196] Started watching for new ooms in manager
Jul 20 09:56:26.587839 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.587796    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 20 09:56:26.588007 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.587995    2580 manager.go:349] Starting recovery of all containers
Jul 20 09:56:26.593649 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.593601    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 20 09:56:26.598559 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.598537    2580 manager.go:354] Recovery completed
Jul 20 09:56:26.601257 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.601232    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 20 09:56:26.603257 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.603245    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:26.605150 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.605131    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:26.605224 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.605163    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:26.605224 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.605172    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:26.605721 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.605707    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 20 09:56:26.605721 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.605719    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 20 09:56:26.605814 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.605709    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:26.605814 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.605733    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 20 09:56:26.606379 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.606366    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:26.606444 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.606389    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:26.606444 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.606399    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:26.607004 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.606992    2580 policy_none.go:49] "None policy: Start"
Jul 20 09:56:26.607035 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.607009    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 20 09:56:26.607035 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.607018    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 20 09:56:26.608082 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.608071    2580 policy_none.go:47] "Start"
Jul 20 09:56:26.613031 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.612966    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-160.ec2.internal.18c3f6efab953eab  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-160.ec2.internal,UID:ip-10-0-133-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-160.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-160.ec2.internal,},FirstTimestamp:2026-07-20 09:56:26.605149867 +0000 UTC m=+0.366750430,LastTimestamp:2026-07-20 09:56:26.605149867 +0000 UTC m=+0.366750430,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-160.ec2.internal,}"
Jul 20 09:56:26.620981 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.620922    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-160.ec2.internal.18c3f6efab958253  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-160.ec2.internal,UID:ip-10-0-133-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-160.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-160.ec2.internal,},FirstTimestamp:2026-07-20 09:56:26.605167187 +0000 UTC m=+0.366767747,LastTimestamp:2026-07-20 09:56:26.605167187 +0000 UTC m=+0.366767747,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-160.ec2.internal,}"
Jul 20 09:56:26.621867 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.621846    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vrxgb"
Jul 20 09:56:26.624629 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.624571    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-160.ec2.internal.18c3f6efab95a50d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-160.ec2.internal,UID:ip-10-0-133-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-160.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-160.ec2.internal,},FirstTimestamp:2026-07-20 09:56:26.605176077 +0000 UTC m=+0.366776637,LastTimestamp:2026-07-20 09:56:26.605176077 +0000 UTC m=+0.366776637,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-160.ec2.internal,}"
Jul 20 09:56:26.632342 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.632254    2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-160.ec2.internal.18c3f6efab953eab\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-160.ec2.internal.18c3f6efab953eab  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-160.ec2.internal,UID:ip-10-0-133-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-160.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-160.ec2.internal,},FirstTimestamp:2026-07-20 09:56:26.605149867 +0000 UTC m=+0.366750430,LastTimestamp:2026-07-20 09:56:26.60637882 +0000 UTC m=+0.367979381,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-160.ec2.internal,}"
Jul 20 09:56:26.632555 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.632517    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vrxgb"
Jul 20 09:56:26.645093 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.645078    2580 manager.go:337] "Starting Device Plugin manager"
Jul 20 09:56:26.660536 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.645125    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 20 09:56:26.660536 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.645139    2580 server.go:87] "Starting device plugin registration server"
Jul 20 09:56:26.660536 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.645423    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 20 09:56:26.660536 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.645452    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 20 09:56:26.660536 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.645604    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 20 09:56:26.660536 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.645678    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 20 09:56:26.660536 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.645686    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 20 09:56:26.660536 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.646136    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 20 09:56:26.660536 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.646182    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 20 09:56:26.692022 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.691994    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 20 09:56:26.692127 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.692039    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 20 09:56:26.692127 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.692059    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 20 09:56:26.692127 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.692094    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 20 09:56:26.694764 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.694748    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:56:26.745996 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.745944    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:26.747051 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.747028    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:26.747175 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.747060    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:26.747175 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.747069    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:26.747175 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.747090    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-160.ec2.internal"
Jul 20 09:56:26.755448 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.755418    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-160.ec2.internal"
Jul 20 09:56:26.755531 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.755456    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-160.ec2.internal\": node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 20 09:56:26.783847 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.783827    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 20 09:56:26.793085 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.793067    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal"]
Jul 20 09:56:26.793148 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.793122    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:26.793966 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.793950    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:26.794038 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.793979    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:26.794038 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.793991    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:26.795237 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.795223    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:26.795392 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.795378    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal"
Jul 20 09:56:26.795462 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.795406    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:26.795913 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.795894    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:26.796011 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.795896    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:26.796011 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.795949    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:26.796011 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.795961    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:26.796011 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.795928    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:26.796011 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.796006    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:26.797132 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.797115    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal"
Jul 20 09:56:26.797132 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.797140    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:26.797777 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.797758    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:26.797857 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.797784    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:26.797857 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.797794    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:26.812094 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.811935    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-160.ec2.internal\" not found" node="ip-10-0-133-160.ec2.internal"
Jul 20 09:56:26.815775 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.815759    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-160.ec2.internal\" not found" node="ip-10-0-133-160.ec2.internal"
Jul 20 09:56:26.884587 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.884560    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 20 09:56:26.886794 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.886778    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99f4219d4f389076d8f30c1cc6d06560-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal\" (UID: \"99f4219d4f389076d8f30c1cc6d06560\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal"
Jul 20 09:56:26.886848 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.886804    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/093884edf7fa6418bd467973eecef80e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-160.ec2.internal\" (UID: \"093884edf7fa6418bd467973eecef80e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal"
Jul 20 09:56:26.886889 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.886822    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99f4219d4f389076d8f30c1cc6d06560-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal\" (UID: \"99f4219d4f389076d8f30c1cc6d06560\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal"
Jul 20 09:56:26.984665 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:26.984634    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 20 09:56:26.987894 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.987872    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99f4219d4f389076d8f30c1cc6d06560-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal\" (UID: \"99f4219d4f389076d8f30c1cc6d06560\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal"
Jul 20 09:56:26.987957 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.987903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99f4219d4f389076d8f30c1cc6d06560-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal\" (UID: \"99f4219d4f389076d8f30c1cc6d06560\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal"
Jul 20 09:56:26.987957 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.987920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/093884edf7fa6418bd467973eecef80e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-160.ec2.internal\" (UID: \"093884edf7fa6418bd467973eecef80e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal"
Jul 20 09:56:26.988027 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.987962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/093884edf7fa6418bd467973eecef80e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-160.ec2.internal\" (UID: \"093884edf7fa6418bd467973eecef80e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal"
Jul 20 09:56:26.988027 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.987971    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99f4219d4f389076d8f30c1cc6d06560-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal\" (UID: \"99f4219d4f389076d8f30c1cc6d06560\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal"
Jul 20 09:56:26.988027 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:26.987982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99f4219d4f389076d8f30c1cc6d06560-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal\" (UID: \"99f4219d4f389076d8f30c1cc6d06560\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal"
Jul 20 09:56:27.085350 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:27.085287    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 20 09:56:27.115636 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:27.115612    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal"
Jul 20 09:56:27.119587 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:27.119392    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal"
Jul 20 09:56:27.185479 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:27.185425    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 20 09:56:27.285917 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:27.285891    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 20 09:56:27.386422 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:27.386351    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 20 09:56:27.486802 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:27.486771    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 20 09:56:27.522097 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:27.522072    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 20 09:56:27.522538 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:27.522207    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 20 09:56:27.558266 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:27.558100    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:56:27.587336 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:27.587311    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 20 09:56:27.625537 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:27.625507    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093884edf7fa6418bd467973eecef80e.slice/crio-c8a11619462a173c3e4e4c0ad80267d1d1679d6e842a44b660dc0455fd5e1f24 WatchSource:0}: Error finding container c8a11619462a173c3e4e4c0ad80267d1d1679d6e842a44b660dc0455fd5e1f24: Status 404 returned error can't find the container with id c8a11619462a173c3e4e4c0ad80267d1d1679d6e842a44b660dc0455fd5e1f24
Jul 20 09:56:27.625772 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:27.625745    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f4219d4f389076d8f30c1cc6d06560.slice/crio-aa510a1b448ebe65f8afde5e7f747d51120132bff6741bcbdc60d51fccd750b1 WatchSource:0}: Error finding container aa510a1b448ebe65f8afde5e7f747d51120132bff6741bcbdc60d51fccd750b1: Status 404 returned error can't find the container with id aa510a1b448ebe65f8afde5e7f747d51120132bff6741bcbdc60d51fccd750b1
Jul 20 09:56:27.629805 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:27.629792    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 09:56:27.636106 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:27.636073    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 09:51:28 +0000 UTC" deadline="2028-02-18 07:42:48.986398821 +0000 UTC"
Jul 20 09:56:27.636106 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:27.636106    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13869h46m21.350296453s"
Jul 20 09:56:27.679306 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:27.679249    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:56:27.688226 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:27.688207    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 20 09:56:27.695390 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:27.695350    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal" event={"ID":"093884edf7fa6418bd467973eecef80e","Type":"ContainerStarted","Data":"c8a11619462a173c3e4e4c0ad80267d1d1679d6e842a44b660dc0455fd5e1f24"}
Jul 20 09:56:27.696316 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:27.696296    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal" event={"ID":"99f4219d4f389076d8f30c1cc6d06560","Type":"ContainerStarted","Data":"aa510a1b448ebe65f8afde5e7f747d51120132bff6741bcbdc60d51fccd750b1"}
Jul 20 09:56:27.742512 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:27.742493    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:27.743349 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:27.743335    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:27.743421 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:27.743366    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:27.743421 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:27.743382    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:27.788864 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:27.788837    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 20 09:56:27.889358 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:27.889327    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 20 09:56:27.989906 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:27.989818    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 20 09:56:27.995038 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:27.995020    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:56:28.086058 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:28.086023    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal"
Jul 20 09:56:28.098698 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:28.098672    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 20 09:56:28.099699 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:28.099675    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal"
Jul 20 09:56:28.108710 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:28.108690    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 20 09:56:30.275248 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.275215    2580 apiserver.go:52] "Watching apiserver"
Jul 20 09:56:30.280755 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.280731    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 20 09:56:30.281923 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.281900    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-42cdx","openshift-dns/node-resolver-jfd48","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal","openshift-multus/network-metrics-daemon-jms2f","openshift-ovn-kubernetes/ovnkube-node-kg8lz","kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2","openshift-cluster-node-tuning-operator/tuned-k9hjp","openshift-image-registry/node-ca-g2gl7","openshift-multus/multus-additional-cni-plugins-ww8gr","openshift-multus/multus-qnvtm","openshift-network-diagnostics/network-check-target-r6xpf","openshift-network-operator/iptables-alerter-w6hbk","kube-system/global-pull-secret-syncer-2tscg"]
Jul 20 09:56:30.283964 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.283943    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.285707 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.285690    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w6hbk"
Jul 20 09:56:30.286485 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.286465    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 20 09:56:30.286576 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.286553    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2k2zq\""
Jul 20 09:56:30.286637 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.286616    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 20 09:56:30.286689 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.286618    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 20 09:56:30.288080 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.288056    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:30.288188 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:30.288140    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:56:30.289769 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.289743    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:30.289862 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:30.289834    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jms2f" podUID="5e4c9043-3e1d-49f1-8550-ce507005bcc4"
Jul 20 09:56:30.290130 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.290113    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 20 09:56:30.290277 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.290139    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:56:30.290615 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.290597    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 20 09:56:30.291017 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.290990    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.291095 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.291080    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9vg2s\""
Jul 20 09:56:30.293915 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.293895    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zbzb8\""
Jul 20 09:56:30.294047 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.294028    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 20 09:56:30.295467 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.295446    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.296020 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.296001    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 20 09:56:30.296101 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.296054    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 20 09:56:30.296190 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.296152    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 20 09:56:30.296365 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.296350    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 20 09:56:30.296899 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.296781    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 20 09:56:30.298236 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.296885    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g2gl7"
Jul 20 09:56:30.298236 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.297454    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:56:30.298236 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.297630    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qnhbr\""
Jul 20 09:56:30.298236 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.298019    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 20 09:56:30.298820 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.298801    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 20 09:56:30.299132 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.299114    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 20 09:56:30.299411 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.299396    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 20 09:56:30.299702 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.299688    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mwbvg\""
Jul 20 09:56:30.299939 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.299921    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 20 09:56:30.300058 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.300045    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-42cdx"
Jul 20 09:56:30.301114 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.301042    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jfd48"
Jul 20 09:56:30.301751 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.301729    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 20 09:56:30.301915 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.301897    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 20 09:56:30.301989 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.301927    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8n5sn\""
Jul 20 09:56:30.303304 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.302739    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.303304 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.303067    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 20 09:56:30.303304 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.303070    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 20 09:56:30.303304 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.303222    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z5h7s\""
Jul 20 09:56:30.304108 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.304092    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.304673 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.304656    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 20 09:56:30.304913 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.304899    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 20 09:56:30.305006 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.304902    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 20 09:56:30.305376 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.305358    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 20 09:56:30.305444 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.305365    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 20 09:56:30.305926 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.305911    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f6vlz\""
Jul 20 09:56:30.305979 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.305945    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hns69\""
Jul 20 09:56:30.305979 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.305963    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 20 09:56:30.306860 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.306839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-run-systemd\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.306930 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.306871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-run-openvswitch\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.306930 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.306895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8fc245f5-c152-44de-ad20-dd240d3741bb-dbus\") pod \"global-pull-secret-syncer-2tscg\" (UID: \"8fc245f5-c152-44de-ad20-dd240d3741bb\") " pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:30.306930 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.306917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqkjn\" (UniqueName: \"kubernetes.io/projected/fc481244-2a08-4f08-a199-7d02c4eaa9fc-kube-api-access-sqkjn\") pod \"node-ca-g2gl7\" (UID: \"fc481244-2a08-4f08-a199-7d02c4eaa9fc\") " pod="openshift-image-registry/node-ca-g2gl7"
Jul 20 09:56:30.307040 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.306966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa43f023-63a0-43ac-bdab-58f78b37370d-host-slash\") pod \"iptables-alerter-w6hbk\" (UID: \"fa43f023-63a0-43ac-bdab-58f78b37370d\") " pod="openshift-network-operator/iptables-alerter-w6hbk"
Jul 20 09:56:30.307040 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.306991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-etc-kubernetes\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.307040 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-lib-modules\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.307040 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4glhv\" (UniqueName: \"kubernetes.io/projected/299ef52a-a423-4f48-8b6b-82e066c00480-kube-api-access-4glhv\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.307275 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.307275 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307212    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rmwz\" (UniqueName: \"kubernetes.io/projected/19c80d0c-2d16-4447-a427-b91f076283f8-kube-api-access-8rmwz\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.307275 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs\") pod \"network-metrics-daemon-jms2f\" (UID: \"5e4c9043-3e1d-49f1-8550-ce507005bcc4\") " pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:30.307275 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/299ef52a-a423-4f48-8b6b-82e066c00480-tmp\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.307275 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc481244-2a08-4f08-a199-7d02c4eaa9fc-host\") pod \"node-ca-g2gl7\" (UID: \"fc481244-2a08-4f08-a199-7d02c4eaa9fc\") " pod="openshift-image-registry/node-ca-g2gl7"
Jul 20 09:56:30.307275 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420-agent-certs\") pod \"konnectivity-agent-42cdx\" (UID: \"35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420\") " pod="kube-system/konnectivity-agent-42cdx"
Jul 20 09:56:30.307519 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307296    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19c80d0c-2d16-4447-a427-b91f076283f8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.307519 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7zv\" (UniqueName: \"kubernetes.io/projected/5e4c9043-3e1d-49f1-8550-ce507005bcc4-kube-api-access-st7zv\") pod \"network-metrics-daemon-jms2f\" (UID: \"5e4c9043-3e1d-49f1-8550-ce507005bcc4\") " pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:30.307519 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307369    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-etc-sysconfig\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.307519 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-host\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.307519 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-run-netns\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.307519 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307445    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa43f023-63a0-43ac-bdab-58f78b37370d-iptables-alerter-script\") pod \"iptables-alerter-w6hbk\" (UID: \"fa43f023-63a0-43ac-bdab-58f78b37370d\") " pod="openshift-network-operator/iptables-alerter-w6hbk"
Jul 20 09:56:30.307519 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9svg\" (UniqueName: \"kubernetes.io/projected/fa43f023-63a0-43ac-bdab-58f78b37370d-kube-api-access-r9svg\") pod \"iptables-alerter-w6hbk\" (UID: \"fa43f023-63a0-43ac-bdab-58f78b37370d\") " pod="openshift-network-operator/iptables-alerter-w6hbk"
Jul 20 09:56:30.307519 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-sys\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.307899 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-etc-openvswitch\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.307899 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxccj\" (UniqueName: \"kubernetes.io/projected/8245a5b2-d969-4790-adb3-c9d22661cc81-kube-api-access-wxccj\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.307899 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-systemd-units\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.307899 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8245a5b2-d969-4790-adb3-c9d22661cc81-ovnkube-config\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.307899 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307662    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8245a5b2-d969-4790-adb3-c9d22661cc81-ovn-node-metrics-cert\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.307899 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19c80d0c-2d16-4447-a427-b91f076283f8-sys-fs\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.307899 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-etc-modprobe-d\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.307899 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307768    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/299ef52a-a423-4f48-8b6b-82e066c00480-etc-tuned\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.307899 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8245a5b2-d969-4790-adb3-c9d22661cc81-env-overrides\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.307899 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-var-lib-kubelet\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.307899 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-slash\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.307899 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-node-log\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.308491 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307914    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-cni-bin\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.308491 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-etc-sysctl-conf\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.308491 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.307979    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-var-lib-openvswitch\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.308491 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-cni-netd\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.308491 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308012    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:30.308491 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308029    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19c80d0c-2d16-4447-a427-b91f076283f8-registration-dir\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.308491 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:30.308067    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6xpf" podUID="1da58eb6-8569-48fb-9e75-ddd8e2e2dcac"
Jul 20 09:56:30.308491 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret\") pod \"global-pull-secret-syncer-2tscg\" (UID: \"8fc245f5-c152-44de-ad20-dd240d3741bb\") " pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:30.308491 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-run\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.308491 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-log-socket\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.308491 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308188    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8245a5b2-d969-4790-adb3-c9d22661cc81-ovnkube-script-lib\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.308491 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8fc245f5-c152-44de-ad20-dd240d3741bb-kubelet-config\") pod \"global-pull-secret-syncer-2tscg\" (UID: \"8fc245f5-c152-44de-ad20-dd240d3741bb\") " pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:30.308491 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308236    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420-konnectivity-ca\") pod \"konnectivity-agent-42cdx\" (UID: \"35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420\") " pod="kube-system/konnectivity-agent-42cdx"
Jul 20 09:56:30.308491 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19c80d0c-2d16-4447-a427-b91f076283f8-device-dir\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.308491 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308294    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19c80d0c-2d16-4447-a427-b91f076283f8-etc-selinux\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.308491 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308315    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 20 09:56:30.308491 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc481244-2a08-4f08-a199-7d02c4eaa9fc-serviceca\") pod \"node-ca-g2gl7\" (UID: \"fc481244-2a08-4f08-a199-7d02c4eaa9fc\") " pod="openshift-image-registry/node-ca-g2gl7"
Jul 20 09:56:30.308982 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-kubelet\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.308982 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-run-ovn\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.308982 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19c80d0c-2d16-4447-a427-b91f076283f8-socket-dir\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.308982 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-etc-sysctl-d\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.308982 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-etc-systemd\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.308982 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.308477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-run-ovn-kubernetes\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.398936 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.398910    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 20 09:56:30.408680 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.408645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/17f16e10-bfe7-4dbf-89c3-9b4d68077838-os-release\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.408836 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.408694    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19c80d0c-2d16-4447-a427-b91f076283f8-sys-fs\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.408836 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.408740    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-etc-modprobe-d\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.408836 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.408791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/299ef52a-a423-4f48-8b6b-82e066c00480-etc-tuned\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.408836 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.408819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17f16e10-bfe7-4dbf-89c3-9b4d68077838-cnibin\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.409068 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.408844    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-cnibin\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.409068 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.408852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19c80d0c-2d16-4447-a427-b91f076283f8-sys-fs\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.409068 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.408970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-host-run-k8s-cni-cncf-io\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.409068 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-etc-modprobe-d\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.409068 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8245a5b2-d969-4790-adb3-c9d22661cc81-env-overrides\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.409068 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-var-lib-kubelet\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.409068 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-slash\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.409384 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-node-log\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.409384 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-cni-bin\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.409384 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409127    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 20 09:56:30.409384 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-slash\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.409384 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409198    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-node-log\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.409384 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-cni-bin\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.409384 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17f16e10-bfe7-4dbf-89c3-9b4d68077838-system-cni-dir\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.409384 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409256    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-var-lib-kubelet\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.409384 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-system-cni-dir\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.409384 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-etc-sysctl-conf\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.409384 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-var-lib-openvswitch\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.409384 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-cni-netd\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.409384 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17f16e10-bfe7-4dbf-89c3-9b4d68077838-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.409384 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/224aadf9-e413-4791-8f19-1950846b8374-cni-binary-copy\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.409384 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-etc-kubernetes\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.409916 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19c80d0c-2d16-4447-a427-b91f076283f8-registration-dir\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.409916 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409441    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-etc-sysctl-conf\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.409916 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret\") pod \"global-pull-secret-syncer-2tscg\" (UID: \"8fc245f5-c152-44de-ad20-dd240d3741bb\") " pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:30.409916 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-run\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.409916 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-log-socket\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.409916 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8245a5b2-d969-4790-adb3-c9d22661cc81-env-overrides\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.409916 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8245a5b2-d969-4790-adb3-c9d22661cc81-ovnkube-script-lib\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.409916 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:30.409529    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:30.409916 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-hostroot\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.409916 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409567    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/224aadf9-e413-4791-8f19-1950846b8374-multus-daemon-config\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.409916 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8fc245f5-c152-44de-ad20-dd240d3741bb-kubelet-config\") pod \"global-pull-secret-syncer-2tscg\" (UID: \"8fc245f5-c152-44de-ad20-dd240d3741bb\") " pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:30.409916 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-var-lib-openvswitch\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.409916 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:30.409636    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret podName:8fc245f5-c152-44de-ad20-dd240d3741bb nodeName:}" failed. No retries permitted until 2026-07-20 09:56:30.909612111 +0000 UTC m=+2.958475751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret") pod "global-pull-secret-syncer-2tscg" (UID: "8fc245f5-c152-44de-ad20-dd240d3741bb") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:30.409916 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409658    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-cni-netd\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.409916 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8fc245f5-c152-44de-ad20-dd240d3741bb-kubelet-config\") pod \"global-pull-secret-syncer-2tscg\" (UID: \"8fc245f5-c152-44de-ad20-dd240d3741bb\") " pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:30.409916 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19c80d0c-2d16-4447-a427-b91f076283f8-registration-dir\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.409916 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420-konnectivity-ca\") pod \"konnectivity-agent-42cdx\" (UID: \"35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420\") " pod="kube-system/konnectivity-agent-42cdx"
Jul 20 09:56:30.410550 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19c80d0c-2d16-4447-a427-b91f076283f8-device-dir\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.410550 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409764    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-run\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.410550 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19c80d0c-2d16-4447-a427-b91f076283f8-etc-selinux\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.410550 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409798    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-log-socket\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.410550 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc481244-2a08-4f08-a199-7d02c4eaa9fc-serviceca\") pod \"node-ca-g2gl7\" (UID: \"fc481244-2a08-4f08-a199-7d02c4eaa9fc\") " pod="openshift-image-registry/node-ca-g2gl7"
Jul 20 09:56:30.410550 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-kubelet\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.410550 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409858    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19c80d0c-2d16-4447-a427-b91f076283f8-device-dir\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.410550 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-run-ovn\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.410550 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17f16e10-bfe7-4dbf-89c3-9b4d68077838-cni-binary-copy\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.410550 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19c80d0c-2d16-4447-a427-b91f076283f8-socket-dir\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.410550 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-etc-sysctl-d\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.410550 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.409987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-etc-systemd\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.410550 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-run-ovn-kubernetes\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.410550 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgcjz\" (UniqueName: \"kubernetes.io/projected/bf6c7a6e-c225-431b-9348-d50084b9d1fe-kube-api-access-wgcjz\") pod \"node-resolver-jfd48\" (UID: \"bf6c7a6e-c225-431b-9348-d50084b9d1fe\") " pod="openshift-dns/node-resolver-jfd48"
Jul 20 09:56:30.410550 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17f16e10-bfe7-4dbf-89c3-9b4d68077838-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.410550 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410106    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-host-run-multus-certs\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.410550 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-run-systemd\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.411297 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-run-openvswitch\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.411297 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-host-var-lib-cni-bin\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.411297 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410243    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vknr\" (UniqueName: \"kubernetes.io/projected/224aadf9-e413-4791-8f19-1950846b8374-kube-api-access-9vknr\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.411297 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc481244-2a08-4f08-a199-7d02c4eaa9fc-serviceca\") pod \"node-ca-g2gl7\" (UID: \"fc481244-2a08-4f08-a199-7d02c4eaa9fc\") " pod="openshift-image-registry/node-ca-g2gl7"
Jul 20 09:56:30.411297 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8fc245f5-c152-44de-ad20-dd240d3741bb-dbus\") pod \"global-pull-secret-syncer-2tscg\" (UID: \"8fc245f5-c152-44de-ad20-dd240d3741bb\") " pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:30.411297 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420-konnectivity-ca\") pod \"konnectivity-agent-42cdx\" (UID: \"35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420\") " pod="kube-system/konnectivity-agent-42cdx"
Jul 20 09:56:30.411297 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqkjn\" (UniqueName: \"kubernetes.io/projected/fc481244-2a08-4f08-a199-7d02c4eaa9fc-kube-api-access-sqkjn\") pod \"node-ca-g2gl7\" (UID: \"fc481244-2a08-4f08-a199-7d02c4eaa9fc\") " pod="openshift-image-registry/node-ca-g2gl7"
Jul 20 09:56:30.411297 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410339    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa43f023-63a0-43ac-bdab-58f78b37370d-host-slash\") pod \"iptables-alerter-w6hbk\" (UID: \"fa43f023-63a0-43ac-bdab-58f78b37370d\") " pod="openshift-network-operator/iptables-alerter-w6hbk"
Jul 20 09:56:30.411297 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410404    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8245a5b2-d969-4790-adb3-c9d22661cc81-ovnkube-script-lib\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.411297 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410479    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-etc-sysctl-d\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.411297 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410502    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19c80d0c-2d16-4447-a427-b91f076283f8-etc-selinux\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.411297 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19c80d0c-2d16-4447-a427-b91f076283f8-socket-dir\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.411297 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9zkm\" (UniqueName: \"kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm\") pod \"network-check-target-r6xpf\" (UID: \"1da58eb6-8569-48fb-9e75-ddd8e2e2dcac\") " pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:30.411297 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa43f023-63a0-43ac-bdab-58f78b37370d-host-slash\") pod \"iptables-alerter-w6hbk\" (UID: \"fa43f023-63a0-43ac-bdab-58f78b37370d\") " pod="openshift-network-operator/iptables-alerter-w6hbk"
Jul 20 09:56:30.411297 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410569    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-run-ovn-kubernetes\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.411297 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-kubelet\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.411297 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410520    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-etc-systemd\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.412092 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-etc-kubernetes\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.412092 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-run-ovn\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.412092 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410623    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-run-systemd\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.412092 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410665    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-etc-kubernetes\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.412092 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-lib-modules\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.412092 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410715    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-run-openvswitch\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.412092 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410742    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4glhv\" (UniqueName: \"kubernetes.io/projected/299ef52a-a423-4f48-8b6b-82e066c00480-kube-api-access-4glhv\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.412092 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49wtc\" (UniqueName: \"kubernetes.io/projected/17f16e10-bfe7-4dbf-89c3-9b4d68077838-kube-api-access-49wtc\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.412092 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410788    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8fc245f5-c152-44de-ad20-dd240d3741bb-dbus\") pod \"global-pull-secret-syncer-2tscg\" (UID: \"8fc245f5-c152-44de-ad20-dd240d3741bb\") " pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:30.412092 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-host-run-netns\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.412092 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-lib-modules\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.412092 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.412092 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rmwz\" (UniqueName: \"kubernetes.io/projected/19c80d0c-2d16-4447-a427-b91f076283f8-kube-api-access-8rmwz\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.412092 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs\") pod \"network-metrics-daemon-jms2f\" (UID: \"5e4c9043-3e1d-49f1-8550-ce507005bcc4\") " pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:30.412092 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/299ef52a-a423-4f48-8b6b-82e066c00480-tmp\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.412092 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.410947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc481244-2a08-4f08-a199-7d02c4eaa9fc-host\") pod \"node-ca-g2gl7\" (UID: \"fc481244-2a08-4f08-a199-7d02c4eaa9fc\") " pod="openshift-image-registry/node-ca-g2gl7"
Jul 20 09:56:30.412092 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411025    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-multus-conf-dir\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.413058 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411052    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420-agent-certs\") pod \"konnectivity-agent-42cdx\" (UID: \"35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420\") " pod="kube-system/konnectivity-agent-42cdx"
Jul 20 09:56:30.413058 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19c80d0c-2d16-4447-a427-b91f076283f8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.413058 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411050    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.413058 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:30.411118    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:30.413058 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7zv\" (UniqueName: \"kubernetes.io/projected/5e4c9043-3e1d-49f1-8550-ce507005bcc4-kube-api-access-st7zv\") pod \"network-metrics-daemon-jms2f\" (UID: \"5e4c9043-3e1d-49f1-8550-ce507005bcc4\") " pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:30.413058 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc481244-2a08-4f08-a199-7d02c4eaa9fc-host\") pod \"node-ca-g2gl7\" (UID: \"fc481244-2a08-4f08-a199-7d02c4eaa9fc\") " pod="openshift-image-registry/node-ca-g2gl7"
Jul 20 09:56:30.413058 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:30.411185    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs podName:5e4c9043-3e1d-49f1-8550-ce507005bcc4 nodeName:}" failed. No retries permitted until 2026-07-20 09:56:30.911148375 +0000 UTC m=+2.960012012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs") pod "network-metrics-daemon-jms2f" (UID: "5e4c9043-3e1d-49f1-8550-ce507005bcc4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:30.413058 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19c80d0c-2d16-4447-a427-b91f076283f8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.413058 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-etc-sysconfig\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.413058 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-host\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.413058 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-etc-sysconfig\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.413058 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf6c7a6e-c225-431b-9348-d50084b9d1fe-tmp-dir\") pod \"node-resolver-jfd48\" (UID: \"bf6c7a6e-c225-431b-9348-d50084b9d1fe\") " pod="openshift-dns/node-resolver-jfd48"
Jul 20 09:56:30.413058 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-os-release\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.413058 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-run-netns\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.413058 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bf6c7a6e-c225-431b-9348-d50084b9d1fe-hosts-file\") pod \"node-resolver-jfd48\" (UID: \"bf6c7a6e-c225-431b-9348-d50084b9d1fe\") " pod="openshift-dns/node-resolver-jfd48"
Jul 20 09:56:30.413058 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-host-var-lib-cni-multus\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.413058 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-host-var-lib-kubelet\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.413933 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411566    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa43f023-63a0-43ac-bdab-58f78b37370d-iptables-alerter-script\") pod \"iptables-alerter-w6hbk\" (UID: \"fa43f023-63a0-43ac-bdab-58f78b37370d\") " pod="openshift-network-operator/iptables-alerter-w6hbk"
Jul 20 09:56:30.413933 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411592    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-multus-socket-dir-parent\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.413933 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9svg\" (UniqueName: \"kubernetes.io/projected/fa43f023-63a0-43ac-bdab-58f78b37370d-kube-api-access-r9svg\") pod \"iptables-alerter-w6hbk\" (UID: \"fa43f023-63a0-43ac-bdab-58f78b37370d\") " pod="openshift-network-operator/iptables-alerter-w6hbk"
Jul 20 09:56:30.413933 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-sys\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.413933 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-host-run-netns\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.413933 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-etc-openvswitch\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.413933 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxccj\" (UniqueName: \"kubernetes.io/projected/8245a5b2-d969-4790-adb3-c9d22661cc81-kube-api-access-wxccj\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.413933 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-etc-openvswitch\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.413933 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411831    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17f16e10-bfe7-4dbf-89c3-9b4d68077838-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.413933 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-multus-cni-dir\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.413933 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.411929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-systemd-units\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.413933 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.412156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8245a5b2-d969-4790-adb3-c9d22661cc81-ovnkube-config\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.413933 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.412200    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8245a5b2-d969-4790-adb3-c9d22661cc81-ovn-node-metrics-cert\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.413933 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.412285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-host\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.413933 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.412831    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/299ef52a-a423-4f48-8b6b-82e066c00480-etc-tuned\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.413933 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.413012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8245a5b2-d969-4790-adb3-c9d22661cc81-systemd-units\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.413933 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.413109    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/299ef52a-a423-4f48-8b6b-82e066c00480-sys\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.414794 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.413411    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa43f023-63a0-43ac-bdab-58f78b37370d-iptables-alerter-script\") pod \"iptables-alerter-w6hbk\" (UID: \"fa43f023-63a0-43ac-bdab-58f78b37370d\") " pod="openshift-network-operator/iptables-alerter-w6hbk"
Jul 20 09:56:30.414794 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.413517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8245a5b2-d969-4790-adb3-c9d22661cc81-ovnkube-config\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.414962 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.414945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8245a5b2-d969-4790-adb3-c9d22661cc81-ovn-node-metrics-cert\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.415308 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.415287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420-agent-certs\") pod \"konnectivity-agent-42cdx\" (UID: \"35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420\") " pod="kube-system/konnectivity-agent-42cdx"
Jul 20 09:56:30.417693 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.417669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/299ef52a-a423-4f48-8b6b-82e066c00480-tmp\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.425781 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.425754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqkjn\" (UniqueName: \"kubernetes.io/projected/fc481244-2a08-4f08-a199-7d02c4eaa9fc-kube-api-access-sqkjn\") pod \"node-ca-g2gl7\" (UID: \"fc481244-2a08-4f08-a199-7d02c4eaa9fc\") " pod="openshift-image-registry/node-ca-g2gl7"
Jul 20 09:56:30.427491 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.427062    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rmwz\" (UniqueName: \"kubernetes.io/projected/19c80d0c-2d16-4447-a427-b91f076283f8-kube-api-access-8rmwz\") pod \"aws-ebs-csi-driver-node-pgjl2\" (UID: \"19c80d0c-2d16-4447-a427-b91f076283f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.427594 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.427544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7zv\" (UniqueName: \"kubernetes.io/projected/5e4c9043-3e1d-49f1-8550-ce507005bcc4-kube-api-access-st7zv\") pod \"network-metrics-daemon-jms2f\" (UID: \"5e4c9043-3e1d-49f1-8550-ce507005bcc4\") " pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:30.427683 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.427661    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9svg\" (UniqueName: \"kubernetes.io/projected/fa43f023-63a0-43ac-bdab-58f78b37370d-kube-api-access-r9svg\") pod \"iptables-alerter-w6hbk\" (UID: \"fa43f023-63a0-43ac-bdab-58f78b37370d\") " pod="openshift-network-operator/iptables-alerter-w6hbk"
Jul 20 09:56:30.427788 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.427696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxccj\" (UniqueName: \"kubernetes.io/projected/8245a5b2-d969-4790-adb3-c9d22661cc81-kube-api-access-wxccj\") pod \"ovnkube-node-kg8lz\" (UID: \"8245a5b2-d969-4790-adb3-c9d22661cc81\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.428329 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.428305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4glhv\" (UniqueName: \"kubernetes.io/projected/299ef52a-a423-4f48-8b6b-82e066c00480-kube-api-access-4glhv\") pod \"tuned-k9hjp\" (UID: \"299ef52a-a423-4f48-8b6b-82e066c00480\") " pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.458072 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.458041    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wl29q"
Jul 20 09:56:30.464318 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.464292    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wl29q"
Jul 20 09:56:30.474593 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.474568    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:56:30.512920 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.512891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-multus-conf-dir\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.513081 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.512930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf6c7a6e-c225-431b-9348-d50084b9d1fe-tmp-dir\") pod \"node-resolver-jfd48\" (UID: \"bf6c7a6e-c225-431b-9348-d50084b9d1fe\") " pod="openshift-dns/node-resolver-jfd48"
Jul 20 09:56:30.513081 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.512953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-os-release\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.513081 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.512974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bf6c7a6e-c225-431b-9348-d50084b9d1fe-hosts-file\") pod \"node-resolver-jfd48\" (UID: \"bf6c7a6e-c225-431b-9348-d50084b9d1fe\") " pod="openshift-dns/node-resolver-jfd48"
Jul 20 09:56:30.513081 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.512994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-host-var-lib-cni-multus\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.513081 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-host-var-lib-kubelet\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.513081 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-multus-conf-dir\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.513081 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-multus-socket-dir-parent\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.513081 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bf6c7a6e-c225-431b-9348-d50084b9d1fe-hosts-file\") pod \"node-resolver-jfd48\" (UID: \"bf6c7a6e-c225-431b-9348-d50084b9d1fe\") " pod="openshift-dns/node-resolver-jfd48"
Jul 20 09:56:30.513081 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513077    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-os-release\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.513486 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17f16e10-bfe7-4dbf-89c3-9b4d68077838-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.513486 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513114    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-host-var-lib-cni-multus\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.513486 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513084    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-multus-socket-dir-parent\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.513486 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513129    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-multus-cni-dir\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.513486 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-host-var-lib-kubelet\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.513486 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/17f16e10-bfe7-4dbf-89c3-9b4d68077838-os-release\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.513486 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-multus-cni-dir\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.513486 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513227    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17f16e10-bfe7-4dbf-89c3-9b4d68077838-cnibin\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.513486 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/17f16e10-bfe7-4dbf-89c3-9b4d68077838-os-release\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.513486 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-cnibin\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.513486 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17f16e10-bfe7-4dbf-89c3-9b4d68077838-cnibin\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.513486 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513278    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-host-run-k8s-cni-cncf-io\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.513486 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf6c7a6e-c225-431b-9348-d50084b9d1fe-tmp-dir\") pod \"node-resolver-jfd48\" (UID: \"bf6c7a6e-c225-431b-9348-d50084b9d1fe\") " pod="openshift-dns/node-resolver-jfd48"
Jul 20 09:56:30.513486 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-host-run-k8s-cni-cncf-io\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.513486 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-cnibin\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.513486 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17f16e10-bfe7-4dbf-89c3-9b4d68077838-system-cni-dir\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.513486 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-system-cni-dir\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.513486 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513384    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17f16e10-bfe7-4dbf-89c3-9b4d68077838-system-cni-dir\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17f16e10-bfe7-4dbf-89c3-9b4d68077838-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-system-cni-dir\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/224aadf9-e413-4791-8f19-1950846b8374-cni-binary-copy\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-etc-kubernetes\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-etc-kubernetes\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-hostroot\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-hostroot\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/224aadf9-e413-4791-8f19-1950846b8374-multus-daemon-config\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17f16e10-bfe7-4dbf-89c3-9b4d68077838-cni-binary-copy\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17f16e10-bfe7-4dbf-89c3-9b4d68077838-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgcjz\" (UniqueName: \"kubernetes.io/projected/bf6c7a6e-c225-431b-9348-d50084b9d1fe-kube-api-access-wgcjz\") pod \"node-resolver-jfd48\" (UID: \"bf6c7a6e-c225-431b-9348-d50084b9d1fe\") " pod="openshift-dns/node-resolver-jfd48"
Jul 20 09:56:30.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17f16e10-bfe7-4dbf-89c3-9b4d68077838-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-host-run-multus-certs\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-host-var-lib-cni-bin\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vknr\" (UniqueName: \"kubernetes.io/projected/224aadf9-e413-4791-8f19-1950846b8374-kube-api-access-9vknr\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zkm\" (UniqueName: \"kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm\") pod \"network-check-target-r6xpf\" (UID: \"1da58eb6-8569-48fb-9e75-ddd8e2e2dcac\") " pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:30.514382 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49wtc\" (UniqueName: \"kubernetes.io/projected/17f16e10-bfe7-4dbf-89c3-9b4d68077838-kube-api-access-49wtc\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.514996 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-host-run-netns\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.514996 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513884    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17f16e10-bfe7-4dbf-89c3-9b4d68077838-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.514996 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.513895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-host-run-netns\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.514996 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.514074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/224aadf9-e413-4791-8f19-1950846b8374-multus-daemon-config\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.514996 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.514080    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17f16e10-bfe7-4dbf-89c3-9b4d68077838-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.514996 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.514089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17f16e10-bfe7-4dbf-89c3-9b4d68077838-cni-binary-copy\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.514996 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.514123    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-host-run-multus-certs\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.514996 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.514135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/224aadf9-e413-4791-8f19-1950846b8374-host-var-lib-cni-bin\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.514996 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.514522    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/224aadf9-e413-4791-8f19-1950846b8374-cni-binary-copy\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.520799 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:30.520332    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:56:30.520799 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:30.520451    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:56:30.520799 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:30.520542    2580 projected.go:196] Error preparing data for projected volume kube-api-access-l9zkm for pod openshift-network-diagnostics/network-check-target-r6xpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:30.520799 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:30.520617    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm podName:1da58eb6-8569-48fb-9e75-ddd8e2e2dcac nodeName:}" failed. No retries permitted until 2026-07-20 09:56:31.020600213 +0000 UTC m=+3.069463838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l9zkm" (UniqueName: "kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm") pod "network-check-target-r6xpf" (UID: "1da58eb6-8569-48fb-9e75-ddd8e2e2dcac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:30.522479 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.522455    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vknr\" (UniqueName: \"kubernetes.io/projected/224aadf9-e413-4791-8f19-1950846b8374-kube-api-access-9vknr\") pod \"multus-qnvtm\" (UID: \"224aadf9-e413-4791-8f19-1950846b8374\") " pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.522649 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.522633    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49wtc\" (UniqueName: \"kubernetes.io/projected/17f16e10-bfe7-4dbf-89c3-9b4d68077838-kube-api-access-49wtc\") pod \"multus-additional-cni-plugins-ww8gr\" (UID: \"17f16e10-bfe7-4dbf-89c3-9b4d68077838\") " pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.527309 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.527259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgcjz\" (UniqueName: \"kubernetes.io/projected/bf6c7a6e-c225-431b-9348-d50084b9d1fe-kube-api-access-wgcjz\") pod \"node-resolver-jfd48\" (UID: \"bf6c7a6e-c225-431b-9348-d50084b9d1fe\") " pod="openshift-dns/node-resolver-jfd48"
Jul 20 09:56:30.596961 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.596923    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2"
Jul 20 09:56:30.606905 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.606881    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w6hbk"
Jul 20 09:56:30.614934 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.614912    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:30.619802 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.619785    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k9hjp"
Jul 20 09:56:30.626906 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.626883    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g2gl7"
Jul 20 09:56:30.632658 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.632641    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-42cdx"
Jul 20 09:56:30.639398 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.639380    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jfd48"
Jul 20 09:56:30.646011 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.645995    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ww8gr"
Jul 20 09:56:30.651722 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.651705    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qnvtm"
Jul 20 09:56:30.917032 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.916949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs\") pod \"network-metrics-daemon-jms2f\" (UID: \"5e4c9043-3e1d-49f1-8550-ce507005bcc4\") " pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:30.917032 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:30.917017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret\") pod \"global-pull-secret-syncer-2tscg\" (UID: \"8fc245f5-c152-44de-ad20-dd240d3741bb\") " pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:30.917245 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:30.917114    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:30.917245 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:30.917118    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:30.917245 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:30.917191    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret podName:8fc245f5-c152-44de-ad20-dd240d3741bb nodeName:}" failed. No retries permitted until 2026-07-20 09:56:31.917155125 +0000 UTC m=+3.966018758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret") pod "global-pull-secret-syncer-2tscg" (UID: "8fc245f5-c152-44de-ad20-dd240d3741bb") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:30.917245 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:30.917214    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs podName:5e4c9043-3e1d-49f1-8550-ce507005bcc4 nodeName:}" failed. No retries permitted until 2026-07-20 09:56:31.917203578 +0000 UTC m=+3.966067216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs") pod "network-metrics-daemon-jms2f" (UID: "5e4c9043-3e1d-49f1-8550-ce507005bcc4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:31.118679 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:31.118644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zkm\" (UniqueName: \"kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm\") pod \"network-check-target-r6xpf\" (UID: \"1da58eb6-8569-48fb-9e75-ddd8e2e2dcac\") " pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:31.118835 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:31.118814    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:56:31.118889 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:31.118842    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:56:31.118889 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:31.118853    2580 projected.go:196] Error preparing data for projected volume kube-api-access-l9zkm for pod openshift-network-diagnostics/network-check-target-r6xpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:31.118951 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:31.118902    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm podName:1da58eb6-8569-48fb-9e75-ddd8e2e2dcac nodeName:}" failed. No retries permitted until 2026-07-20 09:56:32.11888867 +0000 UTC m=+4.167752294 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9zkm" (UniqueName: "kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm") pod "network-check-target-r6xpf" (UID: "1da58eb6-8569-48fb-9e75-ddd8e2e2dcac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:31.253713 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:31.253681    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6c7a6e_c225_431b_9348_d50084b9d1fe.slice/crio-136fa746d5f215909edb8437d3a53136b23204da46c8ee5e0e20edd5cd1b66a1 WatchSource:0}: Error finding container 136fa746d5f215909edb8437d3a53136b23204da46c8ee5e0e20edd5cd1b66a1: Status 404 returned error can't find the container with id 136fa746d5f215909edb8437d3a53136b23204da46c8ee5e0e20edd5cd1b66a1
Jul 20 09:56:31.255244 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:31.255220    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f16e10_bfe7_4dbf_89c3_9b4d68077838.slice/crio-2f13647c1d6102472ad98db0081cc56d41ac52c97bb0fa280dfc4a4bf2d2cc42 WatchSource:0}: Error finding container 2f13647c1d6102472ad98db0081cc56d41ac52c97bb0fa280dfc4a4bf2d2cc42: Status 404 returned error can't find the container with id 2f13647c1d6102472ad98db0081cc56d41ac52c97bb0fa280dfc4a4bf2d2cc42
Jul 20 09:56:31.259406 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:31.259382    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc481244_2a08_4f08_a199_7d02c4eaa9fc.slice/crio-a813bb2466c781a25d92efcd3c52d92a9cee39ee680ba3b322027a8b139dffc6 WatchSource:0}: Error finding container a813bb2466c781a25d92efcd3c52d92a9cee39ee680ba3b322027a8b139dffc6: Status 404 returned error can't find the container with id a813bb2466c781a25d92efcd3c52d92a9cee39ee680ba3b322027a8b139dffc6
Jul 20 09:56:31.260237 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:31.260123    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c80d0c_2d16_4447_a427_b91f076283f8.slice/crio-ede8efa37d4a2ae43bdff89f25de49cf2f5879add4d9cec6efdd3e1e36e3c579 WatchSource:0}: Error finding container ede8efa37d4a2ae43bdff89f25de49cf2f5879add4d9cec6efdd3e1e36e3c579: Status 404 returned error can't find the container with id ede8efa37d4a2ae43bdff89f25de49cf2f5879add4d9cec6efdd3e1e36e3c579
Jul 20 09:56:31.260802 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:31.260781    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cc8bd5_0ca5_4fcf_ab59_61ac94b4d420.slice/crio-ba35fd030416f328a928641c7e20a308c275df60a5f89a8dca0edfff0dbda11a WatchSource:0}: Error finding container ba35fd030416f328a928641c7e20a308c275df60a5f89a8dca0edfff0dbda11a: Status 404 returned error can't find the container with id ba35fd030416f328a928641c7e20a308c275df60a5f89a8dca0edfff0dbda11a
Jul 20 09:56:31.264010 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:31.263985    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa43f023_63a0_43ac_bdab_58f78b37370d.slice/crio-bc36406abc98c0ac62780838183bec4075bcfbb0e6f7d12578ec9a3e8a13a530 WatchSource:0}: Error finding container bc36406abc98c0ac62780838183bec4075bcfbb0e6f7d12578ec9a3e8a13a530: Status 404 returned error can't find the container with id bc36406abc98c0ac62780838183bec4075bcfbb0e6f7d12578ec9a3e8a13a530
Jul 20 09:56:31.265399 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:31.265280    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8245a5b2_d969_4790_adb3_c9d22661cc81.slice/crio-d88871e08ba21a2123c0922556aa1a6a66fc72b8c9026ffdb184ad34e5dd1a36 WatchSource:0}: Error finding container d88871e08ba21a2123c0922556aa1a6a66fc72b8c9026ffdb184ad34e5dd1a36: Status 404 returned error can't find the container with id d88871e08ba21a2123c0922556aa1a6a66fc72b8c9026ffdb184ad34e5dd1a36
Jul 20 09:56:31.267261 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:56:31.267226    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224aadf9_e413_4791_8f19_1950846b8374.slice/crio-7c545b67bfabf999cd70a19a2b4aaef531dbb8a071735600608b89b456d94c68 WatchSource:0}: Error finding container 7c545b67bfabf999cd70a19a2b4aaef531dbb8a071735600608b89b456d94c68: Status 404 returned error can't find the container with id 7c545b67bfabf999cd70a19a2b4aaef531dbb8a071735600608b89b456d94c68
Jul 20 09:56:31.405410 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:31.405372    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:31.405993 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:31.405489    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:56:31.413992 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:31.413964    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2" event={"ID":"19c80d0c-2d16-4447-a427-b91f076283f8","Type":"ContainerStarted","Data":"ede8efa37d4a2ae43bdff89f25de49cf2f5879add4d9cec6efdd3e1e36e3c579"}
Jul 20 09:56:31.415046 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:31.415008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-42cdx" event={"ID":"35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420","Type":"ContainerStarted","Data":"ba35fd030416f328a928641c7e20a308c275df60a5f89a8dca0edfff0dbda11a"}
Jul 20 09:56:31.416216 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:31.416193    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ww8gr" event={"ID":"17f16e10-bfe7-4dbf-89c3-9b4d68077838","Type":"ContainerStarted","Data":"2f13647c1d6102472ad98db0081cc56d41ac52c97bb0fa280dfc4a4bf2d2cc42"}
Jul 20 09:56:31.417324 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:31.417303    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jfd48" event={"ID":"bf6c7a6e-c225-431b-9348-d50084b9d1fe","Type":"ContainerStarted","Data":"136fa746d5f215909edb8437d3a53136b23204da46c8ee5e0e20edd5cd1b66a1"}
Jul 20 09:56:31.419347 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:31.419326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qnvtm" event={"ID":"224aadf9-e413-4791-8f19-1950846b8374","Type":"ContainerStarted","Data":"7c545b67bfabf999cd70a19a2b4aaef531dbb8a071735600608b89b456d94c68"}
Jul 20 09:56:31.420323 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:31.420303    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w6hbk" event={"ID":"fa43f023-63a0-43ac-bdab-58f78b37370d","Type":"ContainerStarted","Data":"bc36406abc98c0ac62780838183bec4075bcfbb0e6f7d12578ec9a3e8a13a530"}
Jul 20 09:56:31.421438 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:31.421418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g2gl7" event={"ID":"fc481244-2a08-4f08-a199-7d02c4eaa9fc","Type":"ContainerStarted","Data":"a813bb2466c781a25d92efcd3c52d92a9cee39ee680ba3b322027a8b139dffc6"}
Jul 20 09:56:31.424126 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:31.424096    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal" event={"ID":"093884edf7fa6418bd467973eecef80e","Type":"ContainerStarted","Data":"d6decd5cde6ce76d6cd14122c158eb49a214e624db1430b82d9828d72da0cb05"}
Jul 20 09:56:31.425577 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:31.425557    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz" event={"ID":"8245a5b2-d969-4790-adb3-c9d22661cc81","Type":"ContainerStarted","Data":"d88871e08ba21a2123c0922556aa1a6a66fc72b8c9026ffdb184ad34e5dd1a36"}
Jul 20 09:56:31.429700 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:31.426958    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k9hjp" event={"ID":"299ef52a-a423-4f48-8b6b-82e066c00480","Type":"ContainerStarted","Data":"c130d14cea5998ab8aa2a9d9f42a13b737cda357f7a717fb6f093f378a1cc973"}
Jul 20 09:56:31.467546 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:31.467502    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 09:51:30 +0000 UTC" deadline="2028-01-10 10:38:51.910934409 +0000 UTC"
Jul 20 09:56:31.467546 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:31.467544    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12936h42m20.443392619s"
Jul 20 09:56:31.925985 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:31.925853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret\") pod \"global-pull-secret-syncer-2tscg\" (UID: \"8fc245f5-c152-44de-ad20-dd240d3741bb\") " pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:31.925985 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:31.925940    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs\") pod \"network-metrics-daemon-jms2f\" (UID: \"5e4c9043-3e1d-49f1-8550-ce507005bcc4\") " pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:31.926235 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:31.926081    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:31.926235 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:31.926148    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs podName:5e4c9043-3e1d-49f1-8550-ce507005bcc4 nodeName:}" failed. No retries permitted until 2026-07-20 09:56:33.926127042 +0000 UTC m=+5.974990672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs") pod "network-metrics-daemon-jms2f" (UID: "5e4c9043-3e1d-49f1-8550-ce507005bcc4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:31.926577 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:31.926554    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:31.926677 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:31.926615    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret podName:8fc245f5-c152-44de-ad20-dd240d3741bb nodeName:}" failed. No retries permitted until 2026-07-20 09:56:33.926598908 +0000 UTC m=+5.975462536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret") pod "global-pull-secret-syncer-2tscg" (UID: "8fc245f5-c152-44de-ad20-dd240d3741bb") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:32.128005 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:32.127967    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zkm\" (UniqueName: \"kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm\") pod \"network-check-target-r6xpf\" (UID: \"1da58eb6-8569-48fb-9e75-ddd8e2e2dcac\") " pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:32.128247 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:32.128215    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:56:32.128247 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:32.128237    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:56:32.128369 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:32.128251    2580 projected.go:196] Error preparing data for projected volume kube-api-access-l9zkm for pod openshift-network-diagnostics/network-check-target-r6xpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:32.128369 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:32.128313    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm podName:1da58eb6-8569-48fb-9e75-ddd8e2e2dcac nodeName:}" failed. No retries permitted until 2026-07-20 09:56:34.12829321 +0000 UTC m=+6.177156858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9zkm" (UniqueName: "kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm") pod "network-check-target-r6xpf" (UID: "1da58eb6-8569-48fb-9e75-ddd8e2e2dcac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:32.408105 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:32.405804    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:32.408105 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:32.405950    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jms2f" podUID="5e4c9043-3e1d-49f1-8550-ce507005bcc4"
Jul 20 09:56:32.408105 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:32.405817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:32.408105 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:32.406049    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6xpf" podUID="1da58eb6-8569-48fb-9e75-ddd8e2e2dcac"
Jul 20 09:56:32.449907 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:32.449869    2580 generic.go:356] "Generic (PLEG): container finished" podID="99f4219d4f389076d8f30c1cc6d06560" containerID="39a481bf856789dcbf9dbcac5412704f0dcaf48d7d72c9ca90e04b66ad588920" exitCode=0
Jul 20 09:56:32.450517 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:32.450490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal" event={"ID":"99f4219d4f389076d8f30c1cc6d06560","Type":"ContainerDied","Data":"39a481bf856789dcbf9dbcac5412704f0dcaf48d7d72c9ca90e04b66ad588920"}
Jul 20 09:56:32.465397 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:32.465329    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal" podStartSLOduration=3.465308417 podStartE2EDuration="3.465308417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:31.442488844 +0000 UTC m=+3.491352490" watchObservedRunningTime="2026-07-20 09:56:32.465308417 +0000 UTC m=+4.514172064"
Jul 20 09:56:32.468368 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:32.468319    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 09:51:30 +0000 UTC" deadline="2028-02-23 07:15:32.911174242 +0000 UTC"
Jul 20 09:56:32.468368 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:32.468347    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13989h19m0.442831454s"
Jul 20 09:56:33.405396 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:33.405366    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:33.405541 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:33.405506    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:56:33.479288 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:33.479251    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal" event={"ID":"99f4219d4f389076d8f30c1cc6d06560","Type":"ContainerStarted","Data":"fb21985bced845b18b200b8572c97a1d44276e688e72aba0ae72113b2f381dfe"}
Jul 20 09:56:33.944882 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:33.944826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret\") pod \"global-pull-secret-syncer-2tscg\" (UID: \"8fc245f5-c152-44de-ad20-dd240d3741bb\") " pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:33.945068 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:33.944906    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs\") pod \"network-metrics-daemon-jms2f\" (UID: \"5e4c9043-3e1d-49f1-8550-ce507005bcc4\") " pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:33.945068 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:33.945030    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:33.945225 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:33.945094    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs podName:5e4c9043-3e1d-49f1-8550-ce507005bcc4 nodeName:}" failed. No retries permitted until 2026-07-20 09:56:37.945074284 +0000 UTC m=+9.993937921 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs") pod "network-metrics-daemon-jms2f" (UID: "5e4c9043-3e1d-49f1-8550-ce507005bcc4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:33.945309 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:33.945258    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:33.945309 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:33.945297    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret podName:8fc245f5-c152-44de-ad20-dd240d3741bb nodeName:}" failed. No retries permitted until 2026-07-20 09:56:37.945285054 +0000 UTC m=+9.994148693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret") pod "global-pull-secret-syncer-2tscg" (UID: "8fc245f5-c152-44de-ad20-dd240d3741bb") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:34.146813 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:34.146772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zkm\" (UniqueName: \"kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm\") pod \"network-check-target-r6xpf\" (UID: \"1da58eb6-8569-48fb-9e75-ddd8e2e2dcac\") " pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:34.146988 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:34.146944    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:56:34.146988 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:34.146965    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:56:34.147086 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:34.147003    2580 projected.go:196] Error preparing data for projected volume kube-api-access-l9zkm for pod openshift-network-diagnostics/network-check-target-r6xpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:34.147086 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:34.147062    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm podName:1da58eb6-8569-48fb-9e75-ddd8e2e2dcac nodeName:}" failed. No retries permitted until 2026-07-20 09:56:38.147043638 +0000 UTC m=+10.195907283 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9zkm" (UniqueName: "kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm") pod "network-check-target-r6xpf" (UID: "1da58eb6-8569-48fb-9e75-ddd8e2e2dcac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:34.406665 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:34.405881    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:34.406665 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:34.406021    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6xpf" podUID="1da58eb6-8569-48fb-9e75-ddd8e2e2dcac"
Jul 20 09:56:34.406665 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:34.406440    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:34.406665 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:34.406578    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jms2f" podUID="5e4c9043-3e1d-49f1-8550-ce507005bcc4"
Jul 20 09:56:35.405493 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:35.405455    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:35.405998 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:35.405590    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:56:36.406386 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:36.406284    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:36.406879 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:36.406388    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6xpf" podUID="1da58eb6-8569-48fb-9e75-ddd8e2e2dcac"
Jul 20 09:56:36.406879 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:36.406422    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:36.406879 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:36.406476    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jms2f" podUID="5e4c9043-3e1d-49f1-8550-ce507005bcc4"
Jul 20 09:56:37.405622 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:37.405541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:37.405770 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:37.405641    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:56:37.978747 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:37.978709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret\") pod \"global-pull-secret-syncer-2tscg\" (UID: \"8fc245f5-c152-44de-ad20-dd240d3741bb\") " pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:37.979137 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:37.978773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs\") pod \"network-metrics-daemon-jms2f\" (UID: \"5e4c9043-3e1d-49f1-8550-ce507005bcc4\") " pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:37.979137 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:37.978884    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:37.979137 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:37.978934    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs podName:5e4c9043-3e1d-49f1-8550-ce507005bcc4 nodeName:}" failed. No retries permitted until 2026-07-20 09:56:45.978921194 +0000 UTC m=+18.027784818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs") pod "network-metrics-daemon-jms2f" (UID: "5e4c9043-3e1d-49f1-8550-ce507005bcc4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:37.979137 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:37.978981    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:37.979137 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:37.979008    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret podName:8fc245f5-c152-44de-ad20-dd240d3741bb nodeName:}" failed. No retries permitted until 2026-07-20 09:56:45.97900151 +0000 UTC m=+18.027865135 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret") pod "global-pull-secret-syncer-2tscg" (UID: "8fc245f5-c152-44de-ad20-dd240d3741bb") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:38.181291 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:38.181248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zkm\" (UniqueName: \"kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm\") pod \"network-check-target-r6xpf\" (UID: \"1da58eb6-8569-48fb-9e75-ddd8e2e2dcac\") " pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:38.181488 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:38.181442    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:56:38.181488 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:38.181468    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:56:38.181488 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:38.181482    2580 projected.go:196] Error preparing data for projected volume kube-api-access-l9zkm for pod openshift-network-diagnostics/network-check-target-r6xpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:38.181647 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:38.181542    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm podName:1da58eb6-8569-48fb-9e75-ddd8e2e2dcac nodeName:}" failed. No retries permitted until 2026-07-20 09:56:46.181522934 +0000 UTC m=+18.230386564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9zkm" (UniqueName: "kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm") pod "network-check-target-r6xpf" (UID: "1da58eb6-8569-48fb-9e75-ddd8e2e2dcac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:38.405280 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:38.405202    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:38.405433 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:38.405351    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jms2f" podUID="5e4c9043-3e1d-49f1-8550-ce507005bcc4"
Jul 20 09:56:38.406346 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:38.405740    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:38.406346 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:38.405835    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6xpf" podUID="1da58eb6-8569-48fb-9e75-ddd8e2e2dcac"
Jul 20 09:56:39.405723 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:39.405677    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:39.406245 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:39.405817    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:56:40.406317 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:40.406284    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:40.406317 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:40.406302    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:40.406753 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:40.406422    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6xpf" podUID="1da58eb6-8569-48fb-9e75-ddd8e2e2dcac"
Jul 20 09:56:40.406753 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:40.406537    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jms2f" podUID="5e4c9043-3e1d-49f1-8550-ce507005bcc4"
Jul 20 09:56:41.405276 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:41.405234    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:41.405443 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:41.405366    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:56:42.405760 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:42.405681    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:42.406180 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:42.405682    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:42.406180 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:42.405814    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jms2f" podUID="5e4c9043-3e1d-49f1-8550-ce507005bcc4"
Jul 20 09:56:42.406180 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:42.405880    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6xpf" podUID="1da58eb6-8569-48fb-9e75-ddd8e2e2dcac"
Jul 20 09:56:43.405403 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:43.405367    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:43.405553 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:43.405514    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:56:44.405496 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:44.405444    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:44.405496 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:44.405477    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:44.405984 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:44.405607    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jms2f" podUID="5e4c9043-3e1d-49f1-8550-ce507005bcc4"
Jul 20 09:56:44.405984 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:44.405732    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6xpf" podUID="1da58eb6-8569-48fb-9e75-ddd8e2e2dcac"
Jul 20 09:56:45.405275 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:45.405237    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:45.405502 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:45.405373    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:56:46.042248 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:46.042214    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs\") pod \"network-metrics-daemon-jms2f\" (UID: \"5e4c9043-3e1d-49f1-8550-ce507005bcc4\") " pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:46.042715 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:46.042269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret\") pod \"global-pull-secret-syncer-2tscg\" (UID: \"8fc245f5-c152-44de-ad20-dd240d3741bb\") " pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:46.042715 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:46.042360    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:46.042715 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:46.042364    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:46.042715 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:46.042433    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret podName:8fc245f5-c152-44de-ad20-dd240d3741bb nodeName:}" failed. No retries permitted until 2026-07-20 09:57:02.042399582 +0000 UTC m=+34.091263208 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret") pod "global-pull-secret-syncer-2tscg" (UID: "8fc245f5-c152-44de-ad20-dd240d3741bb") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:46.042715 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:46.042448    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs podName:5e4c9043-3e1d-49f1-8550-ce507005bcc4 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:02.04244075 +0000 UTC m=+34.091304374 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs") pod "network-metrics-daemon-jms2f" (UID: "5e4c9043-3e1d-49f1-8550-ce507005bcc4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:46.243895 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:46.243857    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zkm\" (UniqueName: \"kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm\") pod \"network-check-target-r6xpf\" (UID: \"1da58eb6-8569-48fb-9e75-ddd8e2e2dcac\") " pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:46.244075 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:46.244041    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:56:46.244075 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:46.244068    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:56:46.244156 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:46.244080    2580 projected.go:196] Error preparing data for projected volume kube-api-access-l9zkm for pod openshift-network-diagnostics/network-check-target-r6xpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:46.244156 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:46.244151    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm podName:1da58eb6-8569-48fb-9e75-ddd8e2e2dcac nodeName:}" failed. No retries permitted until 2026-07-20 09:57:02.244131126 +0000 UTC m=+34.292994766 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9zkm" (UniqueName: "kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm") pod "network-check-target-r6xpf" (UID: "1da58eb6-8569-48fb-9e75-ddd8e2e2dcac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:46.405153 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:46.405065    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:46.405335 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:46.405065    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:46.405335 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:46.405238    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jms2f" podUID="5e4c9043-3e1d-49f1-8550-ce507005bcc4"
Jul 20 09:56:46.405430 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:46.405370    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6xpf" podUID="1da58eb6-8569-48fb-9e75-ddd8e2e2dcac"
Jul 20 09:56:47.404992 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:47.404957    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:47.405488 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:47.405089    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:56:48.406491 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:48.406461    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:48.406966 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:48.406535    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:48.406966 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:48.406613    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6xpf" podUID="1da58eb6-8569-48fb-9e75-ddd8e2e2dcac"
Jul 20 09:56:48.406966 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:48.406753    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jms2f" podUID="5e4c9043-3e1d-49f1-8550-ce507005bcc4"
Jul 20 09:56:49.405204 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:49.405149    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:49.405394 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:49.405334    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:56:50.405978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:50.405934    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:50.405978 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:50.405967    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:50.406391 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:50.406069    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jms2f" podUID="5e4c9043-3e1d-49f1-8550-ce507005bcc4"
Jul 20 09:56:50.406391 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:50.406208    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6xpf" podUID="1da58eb6-8569-48fb-9e75-ddd8e2e2dcac"
Jul 20 09:56:51.405856 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.405552    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:51.405981 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:51.405934    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:56:51.519350 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.519316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz" event={"ID":"8245a5b2-d969-4790-adb3-c9d22661cc81","Type":"ContainerStarted","Data":"ca0a8e6038c2f323cecaa5c88f347561f468b344b7f38235cf8ebed2d7ea0f1a"}
Jul 20 09:56:51.520143 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.519394    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz" event={"ID":"8245a5b2-d969-4790-adb3-c9d22661cc81","Type":"ContainerStarted","Data":"fac3937cf7668026a15ea5985871709be5b55cf4985b5cb7ff956c2eeb76b5a4"}
Jul 20 09:56:51.520143 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.519430    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz" event={"ID":"8245a5b2-d969-4790-adb3-c9d22661cc81","Type":"ContainerStarted","Data":"ab2bb5f08fede886e0e017380f05cfaa4c8d97d7b001b04f973f1245175c7df6"}
Jul 20 09:56:51.520143 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.519469    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz" event={"ID":"8245a5b2-d969-4790-adb3-c9d22661cc81","Type":"ContainerStarted","Data":"ff453ec673d88a16f39b87745b8934b2d61d84865e62d8ad864f8d7c055bb7a6"}
Jul 20 09:56:51.520143 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.519501    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz" event={"ID":"8245a5b2-d969-4790-adb3-c9d22661cc81","Type":"ContainerStarted","Data":"5bae357f48669df59d3da18873a63ceeb80fa70726d65d46d2d4a3a9440ad0a9"}
Jul 20 09:56:51.520143 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.519534    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz" event={"ID":"8245a5b2-d969-4790-adb3-c9d22661cc81","Type":"ContainerStarted","Data":"c986d0f61ebfa481734b1222e4457d9be0445bc653b5fd5022e0438659718395"}
Jul 20 09:56:51.521769 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.521739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k9hjp" event={"ID":"299ef52a-a423-4f48-8b6b-82e066c00480","Type":"ContainerStarted","Data":"407a1b2feb8291db4c9cced3918aa22b6c3c798fefde1287b499eca3d9a0188e"}
Jul 20 09:56:51.523580 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.523553    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2" event={"ID":"19c80d0c-2d16-4447-a427-b91f076283f8","Type":"ContainerStarted","Data":"37ea75886d82a14fd4cc38744694484ae002f76a5c43515d0dd6f09896b0f390"}
Jul 20 09:56:51.525748 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.525487    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-42cdx" event={"ID":"35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420","Type":"ContainerStarted","Data":"bf759f50e07908690a0f19de7514b3b28622f201284d9b62f7ab1c00cb9910d8"}
Jul 20 09:56:51.527496 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.527469    2580 generic.go:356] "Generic (PLEG): container finished" podID="17f16e10-bfe7-4dbf-89c3-9b4d68077838" containerID="3ad68ded21771bab8190e7722768bec55fc8c54ebc614594899cefe5e25cda48" exitCode=0
Jul 20 09:56:51.527622 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.527556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ww8gr" event={"ID":"17f16e10-bfe7-4dbf-89c3-9b4d68077838","Type":"ContainerDied","Data":"3ad68ded21771bab8190e7722768bec55fc8c54ebc614594899cefe5e25cda48"}
Jul 20 09:56:51.529442 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.529417    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jfd48" event={"ID":"bf6c7a6e-c225-431b-9348-d50084b9d1fe","Type":"ContainerStarted","Data":"95bedf6f1ed4df257014c4b85b3bd16c8d3e9757af597069653e4c52c3d54081"}
Jul 20 09:56:51.531775 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.531750    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qnvtm" event={"ID":"224aadf9-e413-4791-8f19-1950846b8374","Type":"ContainerStarted","Data":"b640df87904226d44a9501ec4909063f3921b9d38a177faaf1cbbf18c16367bc"}
Jul 20 09:56:51.533635 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.533614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g2gl7" event={"ID":"fc481244-2a08-4f08-a199-7d02c4eaa9fc","Type":"ContainerStarted","Data":"b97f25749325301a299995c51849c8e0a68ed7f61683bf59f830eb69a8d28700"}
Jul 20 09:56:51.543232 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.543190    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k9hjp" podStartSLOduration=4.110666577 podStartE2EDuration="23.543157131s" totalImagesPullingTime="19.432490554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:51.54291791 +0000 UTC m=+23.591781550" watchObservedRunningTime="2026-07-20 09:56:51.543157131 +0000 UTC m=+23.592020780"
Jul 20 09:56:51.543726 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.543689    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal" podStartSLOduration=22.543677668 podStartE2EDuration="22.543677668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:33.494560229 +0000 UTC m=+5.543423878" watchObservedRunningTime="2026-07-20 09:56:51.543677668 +0000 UTC m=+23.592541316"
Jul 20 09:56:51.647103 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.647042    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qnvtm" podStartSLOduration=4.084620067 podStartE2EDuration="23.647025541s" totalImagesPullingTime="19.562405474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:51.619097396 +0000 UTC m=+23.667961042" watchObservedRunningTime="2026-07-20 09:56:51.647025541 +0000 UTC m=+23.695889188"
Jul 20 09:56:51.678183 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.678111    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g2gl7" podStartSLOduration=4.2424809549999996 podStartE2EDuration="23.678092814s" totalImagesPullingTime="19.435611859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:51.647380269 +0000 UTC m=+23.696243917" watchObservedRunningTime="2026-07-20 09:56:51.678092814 +0000 UTC m=+23.726956486"
Jul 20 09:56:51.711292 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.711231    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-42cdx" podStartSLOduration=12.841095639 podStartE2EDuration="23.711208911s" totalImagesPullingTime="10.870113272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:51.678613019 +0000 UTC m=+23.727476666" watchObservedRunningTime="2026-07-20 09:56:51.711208911 +0000 UTC m=+23.760072563"
Jul 20 09:56:51.711489 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:51.711339    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jfd48" podStartSLOduration=4.271275301 podStartE2EDuration="23.711330121s" totalImagesPullingTime="19.44005482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:51.710807576 +0000 UTC m=+23.759671225" watchObservedRunningTime="2026-07-20 09:56:51.711330121 +0000 UTC m=+23.760193769"
Jul 20 09:56:52.203697 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:52.203668    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 20 09:56:52.373000 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:52.372896    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T09:56:52.203691107Z","UUID":"ea3646b5-6727-44ec-bedc-3f996ba00600","Handler":null,"Name":"","Endpoint":""}
Jul 20 09:56:52.374543 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:52.374519    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 20 09:56:52.374662 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:52.374561    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 20 09:56:52.405605 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:52.405572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:52.405812 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:52.405697    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6xpf" podUID="1da58eb6-8569-48fb-9e75-ddd8e2e2dcac"
Jul 20 09:56:52.405812 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:52.405759    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:52.405903 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:52.405870    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jms2f" podUID="5e4c9043-3e1d-49f1-8550-ce507005bcc4"
Jul 20 09:56:52.538627 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:52.538585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2" event={"ID":"19c80d0c-2d16-4447-a427-b91f076283f8","Type":"ContainerStarted","Data":"2f3949b8899c701b35a332cabe19df62fe559244784e64f9a7b8a6758f269ab7"}
Jul 20 09:56:52.540551 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:52.540476    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w6hbk" event={"ID":"fa43f023-63a0-43ac-bdab-58f78b37370d","Type":"ContainerStarted","Data":"4e81e4126ade61144e3751d1f9f57f6b9bf3aa710d11eb6b8899185c9f8ae8f1"}
Jul 20 09:56:52.556252 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:52.556208    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w6hbk" podStartSLOduration=5.160250376 podStartE2EDuration="24.556193203s" totalImagesPullingTime="19.395942827s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:52.555851567 +0000 UTC m=+24.604715214" watchObservedRunningTime="2026-07-20 09:56:52.556193203 +0000 UTC m=+24.605056851"
Jul 20 09:56:53.405460 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:53.405425    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:53.406965 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:53.405889    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:56:53.545845 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:53.545759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2" event={"ID":"19c80d0c-2d16-4447-a427-b91f076283f8","Type":"ContainerStarted","Data":"a45d7f067c33d541fae47638f030a047197502bf5a87b704a3e4ac039870b7ed"}
Jul 20 09:56:54.405536 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:54.405499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:54.405536 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:54.405521    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:54.405753 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:54.405648    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6xpf" podUID="1da58eb6-8569-48fb-9e75-ddd8e2e2dcac"
Jul 20 09:56:54.405792 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:54.405771    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jms2f" podUID="5e4c9043-3e1d-49f1-8550-ce507005bcc4"
Jul 20 09:56:54.552186 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:54.552123    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz" event={"ID":"8245a5b2-d969-4790-adb3-c9d22661cc81","Type":"ContainerStarted","Data":"15fa6f3322c75ec21df13d280531fbfcfb622b8eaacff1686ce88a8106d2163a"}
Jul 20 09:56:55.405564 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:55.405533    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:55.405740 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:55.405647    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:56:55.600873 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:55.600836    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-42cdx"
Jul 20 09:56:55.601498 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:55.601478    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-42cdx"
Jul 20 09:56:55.617726 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:55.617680    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgjl2" podStartSLOduration=6.001244357 podStartE2EDuration="27.617663088s" totalImagesPullingTime="21.616418731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:53.570225136 +0000 UTC m=+25.619088783" watchObservedRunningTime="2026-07-20 09:56:55.617663088 +0000 UTC m=+27.666526737"
Jul 20 09:56:56.405490 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:56.405239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:56.405637 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:56.405273    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:56.405637 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:56.405618    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jms2f" podUID="5e4c9043-3e1d-49f1-8550-ce507005bcc4"
Jul 20 09:56:56.405717 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:56.405636    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6xpf" podUID="1da58eb6-8569-48fb-9e75-ddd8e2e2dcac"
Jul 20 09:56:56.560552 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:56.560516    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz" event={"ID":"8245a5b2-d969-4790-adb3-c9d22661cc81","Type":"ContainerStarted","Data":"1ac767a02cd729b2b3c47f6ecde7cd1e3da1cca3eaaabca9a8cfe8923783b521"}
Jul 20 09:56:56.560779 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:56.560760    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:56.562300 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:56.562271    2580 generic.go:356] "Generic (PLEG): container finished" podID="17f16e10-bfe7-4dbf-89c3-9b4d68077838" containerID="b27b6067cff9697bbc07ed0f1339bcc75cccd4d9df85b2d45e4f5ef9e193551a" exitCode=0
Jul 20 09:56:56.562408 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:56.562351    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ww8gr" event={"ID":"17f16e10-bfe7-4dbf-89c3-9b4d68077838","Type":"ContainerDied","Data":"b27b6067cff9697bbc07ed0f1339bcc75cccd4d9df85b2d45e4f5ef9e193551a"}
Jul 20 09:56:56.562478 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:56.562446    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-42cdx"
Jul 20 09:56:56.562955 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:56.562896    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-42cdx"
Jul 20 09:56:56.576893 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:56.576869    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:56.596048 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:56.595998    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz" podStartSLOduration=9.125199588 podStartE2EDuration="28.59598382s" totalImagesPullingTime="19.470784232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:56.593933066 +0000 UTC m=+28.642796713" watchObservedRunningTime="2026-07-20 09:56:56.59598382 +0000 UTC m=+28.644847497"
Jul 20 09:56:57.405028 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:57.404977    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:57.405816 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:57.405114    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:56:57.564710 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:57.564498    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:57.564944 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:57.564925    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:57.580515 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:57.580487    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:56:57.975477 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:57.975284    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r6xpf"]
Jul 20 09:56:57.975477 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:57.975477    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:57.975694 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:57.975599    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6xpf" podUID="1da58eb6-8569-48fb-9e75-ddd8e2e2dcac"
Jul 20 09:56:57.976496 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:57.976472    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jms2f"]
Jul 20 09:56:57.976602 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:57.976585    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:57.976707 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:57.976689    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jms2f" podUID="5e4c9043-3e1d-49f1-8550-ce507005bcc4"
Jul 20 09:56:57.986062 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:57.986025    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2tscg"]
Jul 20 09:56:57.986198 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:57.986126    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:57.986276 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:57.986256    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:56:58.568038 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:58.568006    2580 generic.go:356] "Generic (PLEG): container finished" podID="17f16e10-bfe7-4dbf-89c3-9b4d68077838" containerID="6f1d3b670fb492817a19b6f7d6c1440f453f21d3de973de2212be53a7f4a10dc" exitCode=0
Jul 20 09:56:58.568416 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:58.568093    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ww8gr" event={"ID":"17f16e10-bfe7-4dbf-89c3-9b4d68077838","Type":"ContainerDied","Data":"6f1d3b670fb492817a19b6f7d6c1440f453f21d3de973de2212be53a7f4a10dc"}
Jul 20 09:56:59.405365 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:59.405289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:56:59.405537 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:59.405289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:56:59.405537 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:59.405395    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6xpf" podUID="1da58eb6-8569-48fb-9e75-ddd8e2e2dcac"
Jul 20 09:56:59.405537 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:59.405468    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jms2f" podUID="5e4c9043-3e1d-49f1-8550-ce507005bcc4"
Jul 20 09:56:59.405537 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:56:59.405289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:56:59.405537 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:56:59.405537    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:57:00.574874 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:00.574838    2580 generic.go:356] "Generic (PLEG): container finished" podID="17f16e10-bfe7-4dbf-89c3-9b4d68077838" containerID="21c44780758142b47a8f4067ad1c919537f065871a2f6f5cb359e50f137f06ba" exitCode=0
Jul 20 09:57:00.575238 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:00.574884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ww8gr" event={"ID":"17f16e10-bfe7-4dbf-89c3-9b4d68077838","Type":"ContainerDied","Data":"21c44780758142b47a8f4067ad1c919537f065871a2f6f5cb359e50f137f06ba"}
Jul 20 09:57:01.405927 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:01.405895    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:57:01.405927 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:01.405932    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:57:01.406199 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:01.405895    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:57:01.406199 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:01.406036    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jms2f" podUID="5e4c9043-3e1d-49f1-8550-ce507005bcc4"
Jul 20 09:57:01.406199 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:01.406094    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6xpf" podUID="1da58eb6-8569-48fb-9e75-ddd8e2e2dcac"
Jul 20 09:57:01.406199 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:01.406194    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:57:02.057428 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:02.057386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret\") pod \"global-pull-secret-syncer-2tscg\" (UID: \"8fc245f5-c152-44de-ad20-dd240d3741bb\") " pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:57:02.057852 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:02.057454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs\") pod \"network-metrics-daemon-jms2f\" (UID: \"5e4c9043-3e1d-49f1-8550-ce507005bcc4\") " pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:57:02.057852 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:02.057544    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:02.057852 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:02.057550    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:02.057852 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:02.057613    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret podName:8fc245f5-c152-44de-ad20-dd240d3741bb nodeName:}" failed. No retries permitted until 2026-07-20 09:57:34.057595678 +0000 UTC m=+66.106459305 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret") pod "global-pull-secret-syncer-2tscg" (UID: "8fc245f5-c152-44de-ad20-dd240d3741bb") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:02.057852 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:02.057632    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs podName:5e4c9043-3e1d-49f1-8550-ce507005bcc4 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:34.057622257 +0000 UTC m=+66.106485887 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs") pod "network-metrics-daemon-jms2f" (UID: "5e4c9043-3e1d-49f1-8550-ce507005bcc4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:02.259417 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:02.259144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zkm\" (UniqueName: \"kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm\") pod \"network-check-target-r6xpf\" (UID: \"1da58eb6-8569-48fb-9e75-ddd8e2e2dcac\") " pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:57:02.259585 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:02.259294    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:02.259585 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:02.259513    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:02.259585 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:02.259528    2580 projected.go:196] Error preparing data for projected volume kube-api-access-l9zkm for pod openshift-network-diagnostics/network-check-target-r6xpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:02.259743 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:02.259597    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm podName:1da58eb6-8569-48fb-9e75-ddd8e2e2dcac nodeName:}" failed. No retries permitted until 2026-07-20 09:57:34.259579746 +0000 UTC m=+66.308443377 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9zkm" (UniqueName: "kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm") pod "network-check-target-r6xpf" (UID: "1da58eb6-8569-48fb-9e75-ddd8e2e2dcac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:03.405050 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.405008    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:57:03.405050 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.405042    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:57:03.405543 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.405040    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:57:03.405543 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:03.405144    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6xpf" podUID="1da58eb6-8569-48fb-9e75-ddd8e2e2dcac"
Jul 20 09:57:03.405543 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:03.405269    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jms2f" podUID="5e4c9043-3e1d-49f1-8550-ce507005bcc4"
Jul 20 09:57:03.405543 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:03.405335    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tscg" podUID="8fc245f5-c152-44de-ad20-dd240d3741bb"
Jul 20 09:57:03.829643 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.827432    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeReady"
Jul 20 09:57:03.829643 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.827671    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 20 09:57:03.880871 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.880837    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66778bc5c4-fn2ml"]
Jul 20 09:57:03.883701 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.883674    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:03.887835 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.887816    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 20 09:57:03.888739 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.888458    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 20 09:57:03.888739 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.888572    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 20 09:57:03.888739 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.888583    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x8ldn\""
Jul 20 09:57:03.905295 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.905262    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66778bc5c4-fn2ml"]
Jul 20 09:57:03.918614 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.918587    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 20 09:57:03.925140 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.924269    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nn2gp"]
Jul 20 09:57:03.926362 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.926342    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2hnhp"]
Jul 20 09:57:03.926506 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.926488    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nn2gp"
Jul 20 09:57:03.929325 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.928113    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:57:03.934020 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.933997    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 20 09:57:03.934300 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.934278    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g9txk\""
Jul 20 09:57:03.934485 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.934466    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 20 09:57:03.935244 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.934670    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 20 09:57:03.935244 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.934866    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n6zqs\""
Jul 20 09:57:03.935244 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.935057    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 20 09:57:03.944219 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.944202    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 20 09:57:03.963651 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.963625    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2hnhp"]
Jul 20 09:57:03.964318 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.964294    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nn2gp"]
Jul 20 09:57:03.971512 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.971491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5cd510a0-1672-4049-9fb2-561f5aa8ea24-ca-trust-extracted\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:03.971632 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.971519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:03.971632 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.971539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-certificates\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:03.971632 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.971574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm6dk\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-kube-api-access-xm6dk\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:03.971797 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.971637    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5cd510a0-1672-4049-9fb2-561f5aa8ea24-image-registry-private-configuration\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:03.971797 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.971658    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5cd510a0-1672-4049-9fb2-561f5aa8ea24-trusted-ca\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:03.971797 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.971681    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5cd510a0-1672-4049-9fb2-561f5aa8ea24-installation-pull-secrets\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:03.971797 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:03.971706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-bound-sa-token\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:04.072340 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.072307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5cd510a0-1672-4049-9fb2-561f5aa8ea24-ca-trust-extracted\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:04.072340 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.072342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:04.072591 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.072369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-certificates\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:04.072591 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:04.072480    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:57:04.072591 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:04.072500    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66778bc5c4-fn2ml: secret "image-registry-tls" not found
Jul 20 09:57:04.072591 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.072498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:57:04.072591 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.072543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm6dk\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-kube-api-access-xm6dk\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:04.072591 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:04.072569    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls podName:5cd510a0-1672-4049-9fb2-561f5aa8ea24 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:04.572550328 +0000 UTC m=+36.621413974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls") pod "image-registry-66778bc5c4-fn2ml" (UID: "5cd510a0-1672-4049-9fb2-561f5aa8ea24") : secret "image-registry-tls" not found
Jul 20 09:57:04.072894 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.072596    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a84537ba-7e16-4295-8b3c-818c578c58fa-cert\") pod \"ingress-canary-nn2gp\" (UID: \"a84537ba-7e16-4295-8b3c-818c578c58fa\") " pod="openshift-ingress-canary/ingress-canary-nn2gp"
Jul 20 09:57:04.072894 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.072637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5cd510a0-1672-4049-9fb2-561f5aa8ea24-image-registry-private-configuration\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:04.072894 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.072669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5cd510a0-1672-4049-9fb2-561f5aa8ea24-trusted-ca\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:04.072894 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.072697    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpbft\" (UniqueName: \"kubernetes.io/projected/8215ffa1-8dfa-470e-be51-643f5adbc99c-kube-api-access-qpbft\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:57:04.072894 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.072718    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5cd510a0-1672-4049-9fb2-561f5aa8ea24-ca-trust-extracted\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:04.072894 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.072719    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwvmk\" (UniqueName: \"kubernetes.io/projected/a84537ba-7e16-4295-8b3c-818c578c58fa-kube-api-access-zwvmk\") pod \"ingress-canary-nn2gp\" (UID: \"a84537ba-7e16-4295-8b3c-818c578c58fa\") " pod="openshift-ingress-canary/ingress-canary-nn2gp"
Jul 20 09:57:04.072894 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.072771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8215ffa1-8dfa-470e-be51-643f5adbc99c-tmp-dir\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:57:04.072894 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.072798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5cd510a0-1672-4049-9fb2-561f5aa8ea24-installation-pull-secrets\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:04.072894 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.072837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-bound-sa-token\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:04.073416 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.072935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-certificates\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:04.073416 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.073021    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8215ffa1-8dfa-470e-be51-643f5adbc99c-config-volume\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:57:04.073709 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.073689    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5cd510a0-1672-4049-9fb2-561f5aa8ea24-trusted-ca\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:04.077140 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.077112    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5cd510a0-1672-4049-9fb2-561f5aa8ea24-image-registry-private-configuration\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:04.077140 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.077134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5cd510a0-1672-4049-9fb2-561f5aa8ea24-installation-pull-secrets\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:04.086655 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.086599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm6dk\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-kube-api-access-xm6dk\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:04.088094 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.088072    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-bound-sa-token\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:04.173907 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.173870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8215ffa1-8dfa-470e-be51-643f5adbc99c-config-volume\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:57:04.174097 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.173966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:57:04.174097 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.174000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a84537ba-7e16-4295-8b3c-818c578c58fa-cert\") pod \"ingress-canary-nn2gp\" (UID: \"a84537ba-7e16-4295-8b3c-818c578c58fa\") " pod="openshift-ingress-canary/ingress-canary-nn2gp"
Jul 20 09:57:04.174097 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.174042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbft\" (UniqueName: \"kubernetes.io/projected/8215ffa1-8dfa-470e-be51-643f5adbc99c-kube-api-access-qpbft\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:57:04.174097 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.174065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwvmk\" (UniqueName: \"kubernetes.io/projected/a84537ba-7e16-4295-8b3c-818c578c58fa-kube-api-access-zwvmk\") pod \"ingress-canary-nn2gp\" (UID: \"a84537ba-7e16-4295-8b3c-818c578c58fa\") " pod="openshift-ingress-canary/ingress-canary-nn2gp"
Jul 20 09:57:04.174097 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.174088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8215ffa1-8dfa-470e-be51-643f5adbc99c-tmp-dir\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:57:04.174377 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:04.174124    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:04.174377 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:04.174218    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls podName:8215ffa1-8dfa-470e-be51-643f5adbc99c nodeName:}" failed. No retries permitted until 2026-07-20 09:57:04.674197405 +0000 UTC m=+36.723061029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls") pod "dns-default-2hnhp" (UID: "8215ffa1-8dfa-470e-be51-643f5adbc99c") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:04.174474 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.174385    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8215ffa1-8dfa-470e-be51-643f5adbc99c-tmp-dir\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:57:04.174545 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.174524    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8215ffa1-8dfa-470e-be51-643f5adbc99c-config-volume\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:57:04.176854 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.176828    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a84537ba-7e16-4295-8b3c-818c578c58fa-cert\") pod \"ingress-canary-nn2gp\" (UID: \"a84537ba-7e16-4295-8b3c-818c578c58fa\") " pod="openshift-ingress-canary/ingress-canary-nn2gp"
Jul 20 09:57:04.191053 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.191027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpbft\" (UniqueName: \"kubernetes.io/projected/8215ffa1-8dfa-470e-be51-643f5adbc99c-kube-api-access-qpbft\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:57:04.195950 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.195929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwvmk\" (UniqueName: \"kubernetes.io/projected/a84537ba-7e16-4295-8b3c-818c578c58fa-kube-api-access-zwvmk\") pod \"ingress-canary-nn2gp\" (UID: \"a84537ba-7e16-4295-8b3c-818c578c58fa\") " pod="openshift-ingress-canary/ingress-canary-nn2gp"
Jul 20 09:57:04.238515 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.238474    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nn2gp"
Jul 20 09:57:04.577457 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.577425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:04.577971 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:04.577572    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:57:04.577971 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:04.577595    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66778bc5c4-fn2ml: secret "image-registry-tls" not found
Jul 20 09:57:04.577971 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:04.577663    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls podName:5cd510a0-1672-4049-9fb2-561f5aa8ea24 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:05.577645708 +0000 UTC m=+37.626509335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls") pod "image-registry-66778bc5c4-fn2ml" (UID: "5cd510a0-1672-4049-9fb2-561f5aa8ea24") : secret "image-registry-tls" not found
Jul 20 09:57:04.678596 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.678549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:57:04.678791 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:04.678727    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:04.678858 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:04.678799    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls podName:8215ffa1-8dfa-470e-be51-643f5adbc99c nodeName:}" failed. No retries permitted until 2026-07-20 09:57:05.678783849 +0000 UTC m=+37.727647477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls") pod "dns-default-2hnhp" (UID: "8215ffa1-8dfa-470e-be51-643f5adbc99c") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:04.920242 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:04.920067    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nn2gp"]
Jul 20 09:57:04.923969 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:57:04.923937    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84537ba_7e16_4295_8b3c_818c578c58fa.slice/crio-54eed1df521eb92918d8a3586a8ccf5d94dd0775c6e99da81467bd463a316ada WatchSource:0}: Error finding container 54eed1df521eb92918d8a3586a8ccf5d94dd0775c6e99da81467bd463a316ada: Status 404 returned error can't find the container with id 54eed1df521eb92918d8a3586a8ccf5d94dd0775c6e99da81467bd463a316ada
Jul 20 09:57:05.405794 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:05.405760    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:57:05.405984 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:05.405880    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:57:05.406308 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:05.406289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:57:05.408995 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:05.408951    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 20 09:57:05.409109 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:05.408994    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 20 09:57:05.409217 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:05.409200    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 20 09:57:05.409299 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:05.409243    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 20 09:57:05.409715 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:05.409687    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-27wd7\""
Jul 20 09:57:05.409879 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:05.409797    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9x8bv\""
Jul 20 09:57:05.585713 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:05.585628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:05.586439 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:05.585789    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:57:05.586439 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:05.585806    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66778bc5c4-fn2ml: secret "image-registry-tls" not found
Jul 20 09:57:05.586439 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:05.585867    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls podName:5cd510a0-1672-4049-9fb2-561f5aa8ea24 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:07.585847941 +0000 UTC m=+39.634711590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls") pod "image-registry-66778bc5c4-fn2ml" (UID: "5cd510a0-1672-4049-9fb2-561f5aa8ea24") : secret "image-registry-tls" not found
Jul 20 09:57:05.590522 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:05.590485    2580 generic.go:356] "Generic (PLEG): container finished" podID="17f16e10-bfe7-4dbf-89c3-9b4d68077838" containerID="12a045505f0cc6172e7ab4c880e9f6e814bf8d649c6aec4b1372f4ccbb867fa7" exitCode=0
Jul 20 09:57:05.590641 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:05.590543    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ww8gr" event={"ID":"17f16e10-bfe7-4dbf-89c3-9b4d68077838","Type":"ContainerDied","Data":"12a045505f0cc6172e7ab4c880e9f6e814bf8d649c6aec4b1372f4ccbb867fa7"}
Jul 20 09:57:05.591891 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:05.591864    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nn2gp" event={"ID":"a84537ba-7e16-4295-8b3c-818c578c58fa","Type":"ContainerStarted","Data":"54eed1df521eb92918d8a3586a8ccf5d94dd0775c6e99da81467bd463a316ada"}
Jul 20 09:57:05.686721 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:05.686690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:57:05.686855 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:05.686834    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:05.686924 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:05.686901    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls podName:8215ffa1-8dfa-470e-be51-643f5adbc99c nodeName:}" failed. No retries permitted until 2026-07-20 09:57:07.686882926 +0000 UTC m=+39.735746552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls") pod "dns-default-2hnhp" (UID: "8215ffa1-8dfa-470e-be51-643f5adbc99c") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:06.599692 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:06.599650    2580 generic.go:356] "Generic (PLEG): container finished" podID="17f16e10-bfe7-4dbf-89c3-9b4d68077838" containerID="780cb9ebf54dd9017bf8de033112468654bca2cac001b7fcac31180f361421ab" exitCode=0
Jul 20 09:57:06.600129 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:06.599724    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ww8gr" event={"ID":"17f16e10-bfe7-4dbf-89c3-9b4d68077838","Type":"ContainerDied","Data":"780cb9ebf54dd9017bf8de033112468654bca2cac001b7fcac31180f361421ab"}
Jul 20 09:57:07.602278 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:07.602244    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:07.602648 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:07.602417    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:57:07.602648 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:07.602438    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66778bc5c4-fn2ml: secret "image-registry-tls" not found
Jul 20 09:57:07.602648 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:07.602520    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls podName:5cd510a0-1672-4049-9fb2-561f5aa8ea24 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:11.602503929 +0000 UTC m=+43.651367557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls") pod "image-registry-66778bc5c4-fn2ml" (UID: "5cd510a0-1672-4049-9fb2-561f5aa8ea24") : secret "image-registry-tls" not found
Jul 20 09:57:07.603655 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:07.603629    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nn2gp" event={"ID":"a84537ba-7e16-4295-8b3c-818c578c58fa","Type":"ContainerStarted","Data":"c0608ca7a5c3bc42d2c1e88d19aa9fd46fe8dcb0485b45d8817b965b3a9a4e2f"}
Jul 20 09:57:07.607047 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:07.607025    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ww8gr" event={"ID":"17f16e10-bfe7-4dbf-89c3-9b4d68077838","Type":"ContainerStarted","Data":"e525e12c3c5d8d3e2a727e53da3adf23aa79b681739cfc47914e3a5e2c55454d"}
Jul 20 09:57:07.622748 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:07.622706    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nn2gp" podStartSLOduration=2.405661659 podStartE2EDuration="4.622692495s" totalImagesPullingTime="2.217030836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:07.622541922 +0000 UTC m=+39.671405575" watchObservedRunningTime="2026-07-20 09:57:07.622692495 +0000 UTC m=+39.671556142"
Jul 20 09:57:07.650573 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:07.650520    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ww8gr" podStartSLOduration=9.254498867 podStartE2EDuration="39.650498758s" totalImagesPullingTime="30.395999891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:07.649230993 +0000 UTC m=+39.698094641" watchObservedRunningTime="2026-07-20 09:57:07.650498758 +0000 UTC m=+39.699362415"
Jul 20 09:57:07.703279 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:07.703240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:57:07.703461 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:07.703436    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:07.703616 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:07.703585    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls podName:8215ffa1-8dfa-470e-be51-643f5adbc99c nodeName:}" failed. No retries permitted until 2026-07-20 09:57:11.703556378 +0000 UTC m=+43.752420002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls") pod "dns-default-2hnhp" (UID: "8215ffa1-8dfa-470e-be51-643f5adbc99c") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:07.810682 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:07.810592    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:07.992665 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:07.992628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:08.976582 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:08.976550    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:09.078814 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:09.078783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:09.179533 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:09.179505    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:09.278922 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:09.278833    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:09.379288 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:09.379246    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:11.229790 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:11.229750    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:11.328048 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:11.328013    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:11.629286 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:11.629192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:11.629461 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:11.629363    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:57:11.629461 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:11.629384    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66778bc5c4-fn2ml: secret "image-registry-tls" not found
Jul 20 09:57:11.629566 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:11.629470    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls podName:5cd510a0-1672-4049-9fb2-561f5aa8ea24 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:19.629447573 +0000 UTC m=+51.678311200 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls") pod "image-registry-66778bc5c4-fn2ml" (UID: "5cd510a0-1672-4049-9fb2-561f5aa8ea24") : secret "image-registry-tls" not found
Jul 20 09:57:11.729576 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:11.729533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:57:11.729751 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:11.729685    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:11.729751 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:11.729750    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls podName:8215ffa1-8dfa-470e-be51-643f5adbc99c nodeName:}" failed. No retries permitted until 2026-07-20 09:57:19.729733495 +0000 UTC m=+51.778597119 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls") pod "dns-default-2hnhp" (UID: "8215ffa1-8dfa-470e-be51-643f5adbc99c") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:19.686565 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:19.686523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:19.687027 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:19.686675    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:57:19.687027 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:19.686694    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66778bc5c4-fn2ml: secret "image-registry-tls" not found
Jul 20 09:57:19.687027 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:19.686750    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls podName:5cd510a0-1672-4049-9fb2-561f5aa8ea24 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:35.686734589 +0000 UTC m=+67.735598214 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls") pod "image-registry-66778bc5c4-fn2ml" (UID: "5cd510a0-1672-4049-9fb2-561f5aa8ea24") : secret "image-registry-tls" not found
Jul 20 09:57:19.787262 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:19.787219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:57:19.787427 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:19.787373    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:19.787465 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:19.787439    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls podName:8215ffa1-8dfa-470e-be51-643f5adbc99c nodeName:}" failed. No retries permitted until 2026-07-20 09:57:35.787422484 +0000 UTC m=+67.836286109 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls") pod "dns-default-2hnhp" (UID: "8215ffa1-8dfa-470e-be51-643f5adbc99c") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:22.817637 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:22.817604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:22.913882 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:22.913855    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:23.007994 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:23.007947    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:23.753227 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:23.753198    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:24.809795 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:24.809761    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:24.910502 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:24.910465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:29.584422 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.584396    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg8lz"
Jul 20 09:57:29.602741 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.602709    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"]
Jul 20 09:57:29.621879 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.621840    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-rm45p"]
Jul 20 09:57:29.622043 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.621968    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"
Jul 20 09:57:29.624521 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.624495    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 20 09:57:29.624521 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.624513    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-s899d\""
Jul 20 09:57:29.625814 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.625794    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 20 09:57:29.628728 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.628690    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 20 09:57:29.630205 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.630185    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 20 09:57:29.633946 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.633922    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv"]
Jul 20 09:57:29.634056 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.634030    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.639764 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.639730    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 20 09:57:29.639882 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.639813    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jjfx2\""
Jul 20 09:57:29.639976 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.639960    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 20 09:57:29.640342 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.640323    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 20 09:57:29.641994 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.641974    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 20 09:57:29.646965 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.646943    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 20 09:57:29.649672 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.649651    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"]
Jul 20 09:57:29.649758 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.649686    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-rm45p"]
Jul 20 09:57:29.649798 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.649784    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv"
Jul 20 09:57:29.650032 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.649995    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv"]
Jul 20 09:57:29.651972 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.651943    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-k7bcq\""
Jul 20 09:57:29.651972 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.651968    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 20 09:57:29.652152 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.651999    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:57:29.652152 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.652043    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 20 09:57:29.656315 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.656292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/944513c3-bdd1-40f5-8ffd-bd72d60e2d50-snapshots\") pod \"insights-operator-555d9d7885-rm45p\" (UID: \"944513c3-bdd1-40f5-8ffd-bd72d60e2d50\") " pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.656433 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.656319    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-7jrnv\" (UID: \"8eb9a42f-a61f-4dcd-a209-eefe5dc24903\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv"
Jul 20 09:57:29.657872 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.656486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44llm\" (UniqueName: \"kubernetes.io/projected/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-kube-api-access-44llm\") pod \"cluster-monitoring-operator-76cf8966f8-rd75s\" (UID: \"4c1f99dc-12d6-4019-ae66-0d06db01e2a2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"
Jul 20 09:57:29.657872 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.657568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rd75s\" (UID: \"4c1f99dc-12d6-4019-ae66-0d06db01e2a2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"
Jul 20 09:57:29.657872 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.657604    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qqx4\" (UniqueName: \"kubernetes.io/projected/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-kube-api-access-5qqx4\") pod \"cluster-samples-operator-79b7c55f87-7jrnv\" (UID: \"8eb9a42f-a61f-4dcd-a209-eefe5dc24903\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv"
Jul 20 09:57:29.657872 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.657645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944513c3-bdd1-40f5-8ffd-bd72d60e2d50-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-rm45p\" (UID: \"944513c3-bdd1-40f5-8ffd-bd72d60e2d50\") " pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.657872 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.657678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/944513c3-bdd1-40f5-8ffd-bd72d60e2d50-serving-cert\") pod \"insights-operator-555d9d7885-rm45p\" (UID: \"944513c3-bdd1-40f5-8ffd-bd72d60e2d50\") " pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.657872 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.657709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vn62\" (UniqueName: \"kubernetes.io/projected/944513c3-bdd1-40f5-8ffd-bd72d60e2d50-kube-api-access-2vn62\") pod \"insights-operator-555d9d7885-rm45p\" (UID: \"944513c3-bdd1-40f5-8ffd-bd72d60e2d50\") " pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.657872 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.657734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-rd75s\" (UID: \"4c1f99dc-12d6-4019-ae66-0d06db01e2a2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"
Jul 20 09:57:29.658836 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.657960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944513c3-bdd1-40f5-8ffd-bd72d60e2d50-service-ca-bundle\") pod \"insights-operator-555d9d7885-rm45p\" (UID: \"944513c3-bdd1-40f5-8ffd-bd72d60e2d50\") " pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.658836 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.658061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/944513c3-bdd1-40f5-8ffd-bd72d60e2d50-tmp\") pod \"insights-operator-555d9d7885-rm45p\" (UID: \"944513c3-bdd1-40f5-8ffd-bd72d60e2d50\") " pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.753183 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.753138    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn"]
Jul 20 09:57:29.759285 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.759252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/944513c3-bdd1-40f5-8ffd-bd72d60e2d50-tmp\") pod \"insights-operator-555d9d7885-rm45p\" (UID: \"944513c3-bdd1-40f5-8ffd-bd72d60e2d50\") " pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.759414 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.759314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/944513c3-bdd1-40f5-8ffd-bd72d60e2d50-snapshots\") pod \"insights-operator-555d9d7885-rm45p\" (UID: \"944513c3-bdd1-40f5-8ffd-bd72d60e2d50\") " pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.759414 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.759334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-7jrnv\" (UID: \"8eb9a42f-a61f-4dcd-a209-eefe5dc24903\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv"
Jul 20 09:57:29.759414 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.759364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44llm\" (UniqueName: \"kubernetes.io/projected/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-kube-api-access-44llm\") pod \"cluster-monitoring-operator-76cf8966f8-rd75s\" (UID: \"4c1f99dc-12d6-4019-ae66-0d06db01e2a2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"
Jul 20 09:57:29.759414 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.759398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rd75s\" (UID: \"4c1f99dc-12d6-4019-ae66-0d06db01e2a2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"
Jul 20 09:57:29.759759 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.759415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qqx4\" (UniqueName: \"kubernetes.io/projected/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-kube-api-access-5qqx4\") pod \"cluster-samples-operator-79b7c55f87-7jrnv\" (UID: \"8eb9a42f-a61f-4dcd-a209-eefe5dc24903\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv"
Jul 20 09:57:29.759759 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.759434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944513c3-bdd1-40f5-8ffd-bd72d60e2d50-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-rm45p\" (UID: \"944513c3-bdd1-40f5-8ffd-bd72d60e2d50\") " pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.759759 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.759456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/944513c3-bdd1-40f5-8ffd-bd72d60e2d50-serving-cert\") pod \"insights-operator-555d9d7885-rm45p\" (UID: \"944513c3-bdd1-40f5-8ffd-bd72d60e2d50\") " pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.759759 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.759479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vn62\" (UniqueName: \"kubernetes.io/projected/944513c3-bdd1-40f5-8ffd-bd72d60e2d50-kube-api-access-2vn62\") pod \"insights-operator-555d9d7885-rm45p\" (UID: \"944513c3-bdd1-40f5-8ffd-bd72d60e2d50\") " pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.759759 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.759500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-rd75s\" (UID: \"4c1f99dc-12d6-4019-ae66-0d06db01e2a2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"
Jul 20 09:57:29.759759 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:29.759501    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 20 09:57:29.759759 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.759536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944513c3-bdd1-40f5-8ffd-bd72d60e2d50-service-ca-bundle\") pod \"insights-operator-555d9d7885-rm45p\" (UID: \"944513c3-bdd1-40f5-8ffd-bd72d60e2d50\") " pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.759759 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:29.759541    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 20 09:57:29.759759 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:29.759584    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-samples-operator-tls podName:8eb9a42f-a61f-4dcd-a209-eefe5dc24903 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:30.259563797 +0000 UTC m=+62.308427446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-7jrnv" (UID: "8eb9a42f-a61f-4dcd-a209-eefe5dc24903") : secret "samples-operator-tls" not found
Jul 20 09:57:29.759759 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:29.759611    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls podName:4c1f99dc-12d6-4019-ae66-0d06db01e2a2 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:30.259593295 +0000 UTC m=+62.308456943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-rd75s" (UID: "4c1f99dc-12d6-4019-ae66-0d06db01e2a2") : secret "cluster-monitoring-operator-tls" not found
Jul 20 09:57:29.759759 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.759670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/944513c3-bdd1-40f5-8ffd-bd72d60e2d50-tmp\") pod \"insights-operator-555d9d7885-rm45p\" (UID: \"944513c3-bdd1-40f5-8ffd-bd72d60e2d50\") " pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.760161 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.760025    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/944513c3-bdd1-40f5-8ffd-bd72d60e2d50-snapshots\") pod \"insights-operator-555d9d7885-rm45p\" (UID: \"944513c3-bdd1-40f5-8ffd-bd72d60e2d50\") " pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.760161 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.760131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944513c3-bdd1-40f5-8ffd-bd72d60e2d50-service-ca-bundle\") pod \"insights-operator-555d9d7885-rm45p\" (UID: \"944513c3-bdd1-40f5-8ffd-bd72d60e2d50\") " pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.760415 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.760398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-rd75s\" (UID: \"4c1f99dc-12d6-4019-ae66-0d06db01e2a2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"
Jul 20 09:57:29.760462 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.760402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944513c3-bdd1-40f5-8ffd-bd72d60e2d50-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-rm45p\" (UID: \"944513c3-bdd1-40f5-8ffd-bd72d60e2d50\") " pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.763563 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.763545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/944513c3-bdd1-40f5-8ffd-bd72d60e2d50-serving-cert\") pod \"insights-operator-555d9d7885-rm45p\" (UID: \"944513c3-bdd1-40f5-8ffd-bd72d60e2d50\") " pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.773212 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.773161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44llm\" (UniqueName: \"kubernetes.io/projected/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-kube-api-access-44llm\") pod \"cluster-monitoring-operator-76cf8966f8-rd75s\" (UID: \"4c1f99dc-12d6-4019-ae66-0d06db01e2a2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"
Jul 20 09:57:29.777977 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.777957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qqx4\" (UniqueName: \"kubernetes.io/projected/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-kube-api-access-5qqx4\") pod \"cluster-samples-operator-79b7c55f87-7jrnv\" (UID: \"8eb9a42f-a61f-4dcd-a209-eefe5dc24903\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv"
Jul 20 09:57:29.781582 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.781558    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vn62\" (UniqueName: \"kubernetes.io/projected/944513c3-bdd1-40f5-8ffd-bd72d60e2d50-kube-api-access-2vn62\") pod \"insights-operator-555d9d7885-rm45p\" (UID: \"944513c3-bdd1-40f5-8ffd-bd72d60e2d50\") " pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.784956 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.784889    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn"
Jul 20 09:57:29.786871 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.786849    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 20 09:57:29.787905 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.787885    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 20 09:57:29.788023 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.788007    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 20 09:57:29.788087 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.788025    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nnkjb\""
Jul 20 09:57:29.788143 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.788093    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:57:29.790894 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.790874    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn"]
Jul 20 09:57:29.846697 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.846613    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:29.860318 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.860292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-5npqn\" (UID: \"1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn"
Jul 20 09:57:29.860437 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.860352    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkg59\" (UniqueName: \"kubernetes.io/projected/1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6-kube-api-access-nkg59\") pod \"kube-storage-version-migrator-operator-7658d9b749-5npqn\" (UID: \"1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn"
Jul 20 09:57:29.860437 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.860405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-5npqn\" (UID: \"1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn"
Jul 20 09:57:29.923478 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.923440    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:29.944394 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.944361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-rm45p"
Jul 20 09:57:29.961133 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.961105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-5npqn\" (UID: \"1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn"
Jul 20 09:57:29.961271 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.961181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkg59\" (UniqueName: \"kubernetes.io/projected/1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6-kube-api-access-nkg59\") pod \"kube-storage-version-migrator-operator-7658d9b749-5npqn\" (UID: \"1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn"
Jul 20 09:57:29.961271 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.961222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-5npqn\" (UID: \"1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn"
Jul 20 09:57:29.961855 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.961822    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-5npqn\" (UID: \"1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn"
Jul 20 09:57:29.963557 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.963539    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-5npqn\" (UID: \"1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn"
Jul 20 09:57:29.969970 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:29.969942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkg59\" (UniqueName: \"kubernetes.io/projected/1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6-kube-api-access-nkg59\") pod \"kube-storage-version-migrator-operator-7658d9b749-5npqn\" (UID: \"1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn"
Jul 20 09:57:30.021665 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:30.021630    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:30.074561 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:30.074520    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-rm45p"]
Jul 20 09:57:30.078573 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:57:30.078552    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944513c3_bdd1_40f5_8ffd_bd72d60e2d50.slice/crio-eba662761fb613b1eb5fe6177487a515fda096e8e7ad4d599e00dd805d5c0182 WatchSource:0}: Error finding container eba662761fb613b1eb5fe6177487a515fda096e8e7ad4d599e00dd805d5c0182: Status 404 returned error can't find the container with id eba662761fb613b1eb5fe6177487a515fda096e8e7ad4d599e00dd805d5c0182
Jul 20 09:57:30.095412 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:30.095390    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn"
Jul 20 09:57:30.223016 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:30.222980    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn"]
Jul 20 09:57:30.226467 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:57:30.226432    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f34d656_cf4c_4b3a_aa22_ba342ba8cbe6.slice/crio-2cf2c305264da044a79a6ede8be70f78bf6eb3cb35dd79bdca3fecdc2c185530 WatchSource:0}: Error finding container 2cf2c305264da044a79a6ede8be70f78bf6eb3cb35dd79bdca3fecdc2c185530: Status 404 returned error can't find the container with id 2cf2c305264da044a79a6ede8be70f78bf6eb3cb35dd79bdca3fecdc2c185530
Jul 20 09:57:30.264256 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:30.264219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-7jrnv\" (UID: \"8eb9a42f-a61f-4dcd-a209-eefe5dc24903\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv"
Jul 20 09:57:30.264256 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:30.264261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rd75s\" (UID: \"4c1f99dc-12d6-4019-ae66-0d06db01e2a2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"
Jul 20 09:57:30.264452 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:30.264366    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 20 09:57:30.264452 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:30.264369    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 20 09:57:30.264452 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:30.264422    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls podName:4c1f99dc-12d6-4019-ae66-0d06db01e2a2 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:31.264405365 +0000 UTC m=+63.313268990 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-rd75s" (UID: "4c1f99dc-12d6-4019-ae66-0d06db01e2a2") : secret "cluster-monitoring-operator-tls" not found
Jul 20 09:57:30.264452 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:30.264435    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-samples-operator-tls podName:8eb9a42f-a61f-4dcd-a209-eefe5dc24903 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:31.264428907 +0000 UTC m=+63.313292532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-7jrnv" (UID: "8eb9a42f-a61f-4dcd-a209-eefe5dc24903") : secret "samples-operator-tls" not found
Jul 20 09:57:30.658596 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:30.658563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn" event={"ID":"1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6","Type":"ContainerStarted","Data":"2cf2c305264da044a79a6ede8be70f78bf6eb3cb35dd79bdca3fecdc2c185530"}
Jul 20 09:57:30.659806 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:30.659773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-rm45p" event={"ID":"944513c3-bdd1-40f5-8ffd-bd72d60e2d50","Type":"ContainerStarted","Data":"eba662761fb613b1eb5fe6177487a515fda096e8e7ad4d599e00dd805d5c0182"}
Jul 20 09:57:31.128375 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:31.128291    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:31.221963 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:31.221928    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:31.272189 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:31.272131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-7jrnv\" (UID: \"8eb9a42f-a61f-4dcd-a209-eefe5dc24903\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv"
Jul 20 09:57:31.272385 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:31.272211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rd75s\" (UID: \"4c1f99dc-12d6-4019-ae66-0d06db01e2a2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"
Jul 20 09:57:31.272479 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:31.272443    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 20 09:57:31.272606 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:31.272539    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls podName:4c1f99dc-12d6-4019-ae66-0d06db01e2a2 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:33.272517479 +0000 UTC m=+65.321381110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-rd75s" (UID: "4c1f99dc-12d6-4019-ae66-0d06db01e2a2") : secret "cluster-monitoring-operator-tls" not found
Jul 20 09:57:31.272606 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:31.272568    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 20 09:57:31.272722 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:31.272630    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-samples-operator-tls podName:8eb9a42f-a61f-4dcd-a209-eefe5dc24903 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:33.272612747 +0000 UTC m=+65.321476379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-7jrnv" (UID: "8eb9a42f-a61f-4dcd-a209-eefe5dc24903") : secret "samples-operator-tls" not found
Jul 20 09:57:32.308136 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:32.308106    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:32.404673 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:32.404640    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:32.620017 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:32.619932    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-gvfp6"]
Jul 20 09:57:32.622302 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:32.622285    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-gvfp6"
Jul 20 09:57:32.624559 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:32.624538    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-4nv9t\""
Jul 20 09:57:32.633825 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:32.633796    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-gvfp6"]
Jul 20 09:57:32.666255 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:32.666215    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn" event={"ID":"1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6","Type":"ContainerStarted","Data":"1e97b38d76b5d915488a7d9088b9852f3f776f26262b1d1b3a63b931fa8435f7"}
Jul 20 09:57:32.667752 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:32.667728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-rm45p" event={"ID":"944513c3-bdd1-40f5-8ffd-bd72d60e2d50","Type":"ContainerStarted","Data":"251bd64b4566680b672803757eabe4f4b2b82a68cb4d1c14224377f5fdcaa756"}
Jul 20 09:57:32.683282 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:32.683241    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn" podStartSLOduration=1.63527869 podStartE2EDuration="3.683225223s" totalImagesPullingTime="2.047946533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:32.682748043 +0000 UTC m=+64.731611752" watchObservedRunningTime="2026-07-20 09:57:32.683225223 +0000 UTC m=+64.732088852"
Jul 20 09:57:32.684639 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:32.684339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcbnt\" (UniqueName: \"kubernetes.io/projected/8ca55206-517b-428a-a16e-71c14f7c7dbd-kube-api-access-mcbnt\") pod \"network-check-source-7bc5b6bf85-gvfp6\" (UID: \"8ca55206-517b-428a-a16e-71c14f7c7dbd\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-gvfp6"
Jul 20 09:57:32.697861 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:32.697817    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-rm45p" podStartSLOduration=1.503515772 podStartE2EDuration="3.697804197s" totalImagesPullingTime="2.194288425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:32.696996356 +0000 UTC m=+64.745860002" watchObservedRunningTime="2026-07-20 09:57:32.697804197 +0000 UTC m=+64.746667844"
Jul 20 09:57:32.785209 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:32.785157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcbnt\" (UniqueName: \"kubernetes.io/projected/8ca55206-517b-428a-a16e-71c14f7c7dbd-kube-api-access-mcbnt\") pod \"network-check-source-7bc5b6bf85-gvfp6\" (UID: \"8ca55206-517b-428a-a16e-71c14f7c7dbd\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-gvfp6"
Jul 20 09:57:32.793031 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:32.793007    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 20 09:57:32.803385 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:32.803363    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 20 09:57:32.814398 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:32.814369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcbnt\" (UniqueName: \"kubernetes.io/projected/8ca55206-517b-428a-a16e-71c14f7c7dbd-kube-api-access-mcbnt\") pod \"network-check-source-7bc5b6bf85-gvfp6\" (UID: \"8ca55206-517b-428a-a16e-71c14f7c7dbd\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-gvfp6"
Jul 20 09:57:32.936328 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:32.936237    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-gvfp6"
Jul 20 09:57:33.058485 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.058456    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-gvfp6"]
Jul 20 09:57:33.062200 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:57:33.062151    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca55206_517b_428a_a16e_71c14f7c7dbd.slice/crio-9510fac23f5691bb4da77c70bb3e474eb58e813ea726dae2158fa5ebe4d20dfe WatchSource:0}: Error finding container 9510fac23f5691bb4da77c70bb3e474eb58e813ea726dae2158fa5ebe4d20dfe: Status 404 returned error can't find the container with id 9510fac23f5691bb4da77c70bb3e474eb58e813ea726dae2158fa5ebe4d20dfe
Jul 20 09:57:33.290726 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.290684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-7jrnv\" (UID: \"8eb9a42f-a61f-4dcd-a209-eefe5dc24903\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv"
Jul 20 09:57:33.290726 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.290730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rd75s\" (UID: \"4c1f99dc-12d6-4019-ae66-0d06db01e2a2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"
Jul 20 09:57:33.291008 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:33.290841    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 20 09:57:33.291008 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:33.290864    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 20 09:57:33.291008 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:33.290894    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls podName:4c1f99dc-12d6-4019-ae66-0d06db01e2a2 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:37.290881215 +0000 UTC m=+69.339744843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-rd75s" (UID: "4c1f99dc-12d6-4019-ae66-0d06db01e2a2") : secret "cluster-monitoring-operator-tls" not found
Jul 20 09:57:33.291008 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:33.290931    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-samples-operator-tls podName:8eb9a42f-a61f-4dcd-a209-eefe5dc24903 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:37.290912797 +0000 UTC m=+69.339776430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-7jrnv" (UID: "8eb9a42f-a61f-4dcd-a209-eefe5dc24903") : secret "samples-operator-tls" not found
Jul 20 09:57:33.654594 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.654501    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:33.672778 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.672739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-gvfp6" event={"ID":"8ca55206-517b-428a-a16e-71c14f7c7dbd","Type":"ContainerStarted","Data":"9510fac23f5691bb4da77c70bb3e474eb58e813ea726dae2158fa5ebe4d20dfe"}
Jul 20 09:57:33.743695 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.743643    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:33.827398 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.827348    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:33.881865 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.880467    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-h8tvt"]
Jul 20 09:57:33.883310 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.883288    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:33.886531 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.886263    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m8wfj\""
Jul 20 09:57:33.886531 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.886277    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 20 09:57:33.886531 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.886506    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 20 09:57:33.901008 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.900918    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h8tvt"]
Jul 20 09:57:33.939376 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.939276    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:33.998577 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.998546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:33.998762 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.998707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/540eee1e-9d83-40b2-afb6-970f4df13724-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:33.998840 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.998771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/540eee1e-9d83-40b2-afb6-970f4df13724-crio-socket\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:33.998840 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.998818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sz2c\" (UniqueName: \"kubernetes.io/projected/540eee1e-9d83-40b2-afb6-970f4df13724-kube-api-access-7sz2c\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:33.998939 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:33.998885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/540eee1e-9d83-40b2-afb6-970f4df13724-data-volume\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:34.039373 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.039322    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:34.099441 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.099389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/540eee1e-9d83-40b2-afb6-970f4df13724-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:34.099441 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.099436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs\") pod \"network-metrics-daemon-jms2f\" (UID: \"5e4c9043-3e1d-49f1-8550-ce507005bcc4\") " pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:57:34.099681 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.099478    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/540eee1e-9d83-40b2-afb6-970f4df13724-crio-socket\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:34.099681 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.099514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret\") pod \"global-pull-secret-syncer-2tscg\" (UID: \"8fc245f5-c152-44de-ad20-dd240d3741bb\") " pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:57:34.099681 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.099543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sz2c\" (UniqueName: \"kubernetes.io/projected/540eee1e-9d83-40b2-afb6-970f4df13724-kube-api-access-7sz2c\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:34.099681 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:34.099546    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:57:34.099681 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.099584    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/540eee1e-9d83-40b2-afb6-970f4df13724-crio-socket\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:34.099681 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.099609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/540eee1e-9d83-40b2-afb6-970f4df13724-data-volume\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:34.099681 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:34.099625    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/540eee1e-9d83-40b2-afb6-970f4df13724-kube-rbac-proxy-cm podName:540eee1e-9d83-40b2-afb6-970f4df13724 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:34.599603929 +0000 UTC m=+66.648467553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/540eee1e-9d83-40b2-afb6-970f4df13724-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h8tvt" (UID: "540eee1e-9d83-40b2-afb6-970f4df13724") : configmap "kube-rbac-proxy" not found
Jul 20 09:57:34.100001 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.099780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:34.100001 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:34.099935    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:34.100091 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.099943    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/540eee1e-9d83-40b2-afb6-970f4df13724-data-volume\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:34.100091 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:34.100075    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls podName:540eee1e-9d83-40b2-afb6-970f4df13724 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:34.599997944 +0000 UTC m=+66.648861571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h8tvt" (UID: "540eee1e-9d83-40b2-afb6-970f4df13724") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:34.101590 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.101567    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 20 09:57:34.101590 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.101586    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 20 09:57:34.109955 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:34.109933    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 20 09:57:34.110075 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:34.110005    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs podName:5e4c9043-3e1d-49f1-8550-ce507005bcc4 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:38.109984283 +0000 UTC m=+130.158847935 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs") pod "network-metrics-daemon-jms2f" (UID: "5e4c9043-3e1d-49f1-8550-ce507005bcc4") : secret "metrics-daemon-secret" not found
Jul 20 09:57:34.112582 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.112564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fc245f5-c152-44de-ad20-dd240d3741bb-original-pull-secret\") pod \"global-pull-secret-syncer-2tscg\" (UID: \"8fc245f5-c152-44de-ad20-dd240d3741bb\") " pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:57:34.113597 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.113579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sz2c\" (UniqueName: \"kubernetes.io/projected/540eee1e-9d83-40b2-afb6-970f4df13724-kube-api-access-7sz2c\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:34.219951 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.219909    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tscg"
Jul 20 09:57:34.301873 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.301840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zkm\" (UniqueName: \"kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm\") pod \"network-check-target-r6xpf\" (UID: \"1da58eb6-8569-48fb-9e75-ddd8e2e2dcac\") " pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:57:34.304854 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.304824    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9zkm\" (UniqueName: \"kubernetes.io/projected/1da58eb6-8569-48fb-9e75-ddd8e2e2dcac-kube-api-access-l9zkm\") pod \"network-check-target-r6xpf\" (UID: \"1da58eb6-8569-48fb-9e75-ddd8e2e2dcac\") " pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:57:34.340795 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.340744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:34.360187 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.360142    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2tscg"]
Jul 20 09:57:34.363379 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:57:34.363320    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc245f5_c152_44de_ad20_dd240d3741bb.slice/crio-23cd5760ddf8b6a941785ea00e408a4ddb2aeb31e30f1f3de4531eab131e669b WatchSource:0}: Error finding container 23cd5760ddf8b6a941785ea00e408a4ddb2aeb31e30f1f3de4531eab131e669b: Status 404 returned error can't find the container with id 23cd5760ddf8b6a941785ea00e408a4ddb2aeb31e30f1f3de4531eab131e669b
Jul 20 09:57:34.440527 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.440488    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:34.529120 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.529034    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-27wd7\""
Jul 20 09:57:34.537900 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.537873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:57:34.604592 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.604562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/540eee1e-9d83-40b2-afb6-970f4df13724-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:34.604721 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:34.604681    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:57:34.604775 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:34.604747    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/540eee1e-9d83-40b2-afb6-970f4df13724-kube-rbac-proxy-cm podName:540eee1e-9d83-40b2-afb6-970f4df13724 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:35.604728975 +0000 UTC m=+67.653592607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/540eee1e-9d83-40b2-afb6-970f4df13724-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h8tvt" (UID: "540eee1e-9d83-40b2-afb6-970f4df13724") : configmap "kube-rbac-proxy" not found
Jul 20 09:57:34.604775 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.604682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:34.604775 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:34.604760    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:34.604925 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:34.604795    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls podName:540eee1e-9d83-40b2-afb6-970f4df13724 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:35.604784414 +0000 UTC m=+67.653648057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h8tvt" (UID: "540eee1e-9d83-40b2-afb6-970f4df13724") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:34.676921 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.676846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2tscg" event={"ID":"8fc245f5-c152-44de-ad20-dd240d3741bb","Type":"ContainerStarted","Data":"23cd5760ddf8b6a941785ea00e408a4ddb2aeb31e30f1f3de4531eab131e669b"}
Jul 20 09:57:34.680967 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:34.680941    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r6xpf"]
Jul 20 09:57:34.684057 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:57:34.684025    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da58eb6_8569_48fb_9e75_ddd8e2e2dcac.slice/crio-045e19e2c1f10124a748152b4ad610f845f49741997339ae00de28ecd2b0fac2 WatchSource:0}: Error finding container 045e19e2c1f10124a748152b4ad610f845f49741997339ae00de28ecd2b0fac2: Status 404 returned error can't find the container with id 045e19e2c1f10124a748152b4ad610f845f49741997339ae00de28ecd2b0fac2
Jul 20 09:57:35.021663 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:35.021625    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:35.086363 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:35.086323    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:35.408738 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:35.408646    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:35.510671 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:35.510636    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:35.613121 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:35.613084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:35.613364 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:35.613199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/540eee1e-9d83-40b2-afb6-970f4df13724-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:35.613364 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:35.613282    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:35.613364 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:35.613292    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:57:35.613364 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:35.613360    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls podName:540eee1e-9d83-40b2-afb6-970f4df13724 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:37.613337214 +0000 UTC m=+69.662200854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h8tvt" (UID: "540eee1e-9d83-40b2-afb6-970f4df13724") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:35.613555 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:35.613383    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/540eee1e-9d83-40b2-afb6-970f4df13724-kube-rbac-proxy-cm podName:540eee1e-9d83-40b2-afb6-970f4df13724 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:37.613372296 +0000 UTC m=+69.662235925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/540eee1e-9d83-40b2-afb6-970f4df13724-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h8tvt" (UID: "540eee1e-9d83-40b2-afb6-970f4df13724") : configmap "kube-rbac-proxy" not found
Jul 20 09:57:35.682194 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:35.682067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r6xpf" event={"ID":"1da58eb6-8569-48fb-9e75-ddd8e2e2dcac","Type":"ContainerStarted","Data":"045e19e2c1f10124a748152b4ad610f845f49741997339ae00de28ecd2b0fac2"}
Jul 20 09:57:35.714557 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:35.714524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:57:35.714756 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:35.714733    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:57:35.714823 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:35.714790    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66778bc5c4-fn2ml: secret "image-registry-tls" not found
Jul 20 09:57:35.714877 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:35.714866    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls podName:5cd510a0-1672-4049-9fb2-561f5aa8ea24 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:07.714842028 +0000 UTC m=+99.763705666 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls") pod "image-registry-66778bc5c4-fn2ml" (UID: "5cd510a0-1672-4049-9fb2-561f5aa8ea24") : secret "image-registry-tls" not found
Jul 20 09:57:35.815065 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:35.815023    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:57:35.815275 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:35.815237    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:35.815340 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:35.815320    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls podName:8215ffa1-8dfa-470e-be51-643f5adbc99c nodeName:}" failed. No retries permitted until 2026-07-20 09:58:07.815297548 +0000 UTC m=+99.864161174 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls") pod "dns-default-2hnhp" (UID: "8215ffa1-8dfa-470e-be51-643f5adbc99c") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:36.170854 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:36.170819    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:36.244158 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:36.244119    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:36.486437 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:36.486412    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jfd48_bf6c7a6e-c225-431b-9348-d50084b9d1fe/dns-node-resolver/0.log"
Jul 20 09:57:36.667400 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:36.667363    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:36.687184 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:36.687130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-gvfp6" event={"ID":"8ca55206-517b-428a-a16e-71c14f7c7dbd","Type":"ContainerStarted","Data":"0c95696fe318459bc3b291061998a2f81164dfe93ba3d30621ea3725d9235231"}
Jul 20 09:57:36.689112 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:36.689086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r6xpf" event={"ID":"1da58eb6-8569-48fb-9e75-ddd8e2e2dcac","Type":"ContainerStarted","Data":"ea883e3d0826330c57f59a5da42d7b1bdb23e9f39fe777d211c9adab16128dde"}
Jul 20 09:57:36.689270 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:36.689236    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:57:36.709437 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:36.709386    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-gvfp6" podStartSLOduration=1.302269852 podStartE2EDuration="4.709366881s" totalImagesPullingTime="3.407097029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:36.709035832 +0000 UTC m=+68.757899490" watchObservedRunningTime="2026-07-20 09:57:36.709366881 +0000 UTC m=+68.758230529"
Jul 20 09:57:36.730022 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:36.729941    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-r6xpf" podStartSLOduration=66.943961048 podStartE2EDuration="1m8.729921564s" totalImagesPullingTime="1.785960516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:36.729348273 +0000 UTC m=+68.778211921" watchObservedRunningTime="2026-07-20 09:57:36.729921564 +0000 UTC m=+68.778785213"
Jul 20 09:57:36.756837 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:36.756798    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:37.291443 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:37.291412    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g2gl7_fc481244-2a08-4f08-a199-7d02c4eaa9fc/node-ca/0.log"
Jul 20 09:57:37.328139 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:37.328097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-7jrnv\" (UID: \"8eb9a42f-a61f-4dcd-a209-eefe5dc24903\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv"
Jul 20 09:57:37.328335 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:37.328153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rd75s\" (UID: \"4c1f99dc-12d6-4019-ae66-0d06db01e2a2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"
Jul 20 09:57:37.328335 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:37.328296    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 20 09:57:37.328335 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:37.328313    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 20 09:57:37.328507 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:37.328374    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-samples-operator-tls podName:8eb9a42f-a61f-4dcd-a209-eefe5dc24903 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:45.328355832 +0000 UTC m=+77.377219462 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-7jrnv" (UID: "8eb9a42f-a61f-4dcd-a209-eefe5dc24903") : secret "samples-operator-tls" not found
Jul 20 09:57:37.328507 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:37.328394    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls podName:4c1f99dc-12d6-4019-ae66-0d06db01e2a2 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:45.328384807 +0000 UTC m=+77.377248432 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-rd75s" (UID: "4c1f99dc-12d6-4019-ae66-0d06db01e2a2") : secret "cluster-monitoring-operator-tls" not found
Jul 20 09:57:37.631855 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:37.631769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:37.632034 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:37.631878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/540eee1e-9d83-40b2-afb6-970f4df13724-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:37.632034 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:37.631951    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:37.632034 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:37.632009    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:57:37.632034 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:37.632027    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls podName:540eee1e-9d83-40b2-afb6-970f4df13724 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:41.632005303 +0000 UTC m=+73.680868934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h8tvt" (UID: "540eee1e-9d83-40b2-afb6-970f4df13724") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:37.632309 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:37.632063    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/540eee1e-9d83-40b2-afb6-970f4df13724-kube-rbac-proxy-cm podName:540eee1e-9d83-40b2-afb6-970f4df13724 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:41.63205133 +0000 UTC m=+73.680914959 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/540eee1e-9d83-40b2-afb6-970f4df13724-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h8tvt" (UID: "540eee1e-9d83-40b2-afb6-970f4df13724") : configmap "kube-rbac-proxy" not found
Jul 20 09:57:37.887009 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:37.886924    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nn2gp_a84537ba-7e16-4295-8b3c-818c578c58fa/serve-healthcheck-canary/0.log"
Jul 20 09:57:38.082390 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:38.082296    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:38.689303 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:38.689221    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-5npqn_1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6/kube-storage-version-migrator-operator/0.log"
Jul 20 09:57:38.702673 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:38.702637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2tscg" event={"ID":"8fc245f5-c152-44de-ad20-dd240d3741bb","Type":"ContainerStarted","Data":"62d9d3f5f98adf967d5913080dbb2e5f5a15ac527baf612f00990b88dd8a68e9"}
Jul 20 09:57:38.728118 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:38.728074    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2tscg" podStartSLOduration=66.72496349 podStartE2EDuration="1m10.728059222s" totalImagesPullingTime="4.003095732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:38.727571015 +0000 UTC m=+70.776434661" watchObservedRunningTime="2026-07-20 09:57:38.728059222 +0000 UTC m=+70.776922862"
Jul 20 09:57:40.685150 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:40.685112    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:40.785963 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:40.785928    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:40.978353 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:40.978323    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:41.134094 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:41.134061    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:41.256754 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:41.256661    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:41.357857 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:41.357825    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:41.461661 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:41.461610    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:41.568230 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:41.568124    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:41.662505 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:41.662469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/540eee1e-9d83-40b2-afb6-970f4df13724-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:41.662670 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:41.662549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:41.662670 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:41.662595    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:57:41.662670 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:41.662656    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:41.662670 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:41.662667    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/540eee1e-9d83-40b2-afb6-970f4df13724-kube-rbac-proxy-cm podName:540eee1e-9d83-40b2-afb6-970f4df13724 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:49.662649384 +0000 UTC m=+81.711513029 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/540eee1e-9d83-40b2-afb6-970f4df13724-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h8tvt" (UID: "540eee1e-9d83-40b2-afb6-970f4df13724") : configmap "kube-rbac-proxy" not found
Jul 20 09:57:41.662807 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:41.662696    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls podName:540eee1e-9d83-40b2-afb6-970f4df13724 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:49.662684674 +0000 UTC m=+81.711548298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h8tvt" (UID: "540eee1e-9d83-40b2-afb6-970f4df13724") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:41.684094 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:41.684056    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:42.865441 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:42.865407    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:43.062108 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:43.062075    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:43.274446 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:43.274414    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:43.472467 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:43.472425    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:43.682104 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:43.682002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:44.342891 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:44.342861    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:44.658024 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:44.657944    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:44.729095 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:44.729052    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:44.979728 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:44.979699    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:45.081460 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:45.081422    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:45.288685 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:45.288593    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:45.397445 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:45.397410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-7jrnv\" (UID: \"8eb9a42f-a61f-4dcd-a209-eefe5dc24903\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv"
Jul 20 09:57:45.397851 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:45.397452    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rd75s\" (UID: \"4c1f99dc-12d6-4019-ae66-0d06db01e2a2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"
Jul 20 09:57:45.397851 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:45.397583    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 20 09:57:45.397851 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:45.397639    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls podName:4c1f99dc-12d6-4019-ae66-0d06db01e2a2 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:01.397624546 +0000 UTC m=+93.446488171 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-rd75s" (UID: "4c1f99dc-12d6-4019-ae66-0d06db01e2a2") : secret "cluster-monitoring-operator-tls" not found
Jul 20 09:57:45.399976 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:45.399958    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8eb9a42f-a61f-4dcd-a209-eefe5dc24903-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-7jrnv\" (UID: \"8eb9a42f-a61f-4dcd-a209-eefe5dc24903\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv"
Jul 20 09:57:45.475920 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:45.475887    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:45.560011 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:45.559925    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv"
Jul 20 09:57:45.661707 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:45.661676    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:45.687391 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:45.687362    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv"]
Jul 20 09:57:45.725709 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:45.725673    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:46.733233 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:46.733195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv" event={"ID":"8eb9a42f-a61f-4dcd-a209-eefe5dc24903","Type":"ContainerStarted","Data":"a5163180230934892b8d45fa783ca0469ac33246395f6bca846d0100e73e8d54"}
Jul 20 09:57:46.842562 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:46.842524    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:46.949545 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:46.949507    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:47.102828 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:47.102728    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:47.258368 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:47.258334    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:47.359442 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:47.359413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:47.477965 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:47.477933    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:47.666981 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:47.666889    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:47.737730 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:47.737702    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv" event={"ID":"8eb9a42f-a61f-4dcd-a209-eefe5dc24903","Type":"ContainerStarted","Data":"6e9f0eb78eacf14a61350fb0b6458cffc445428026d6dc420d48e93acdca0cdd"}
Jul 20 09:57:47.738122 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:47.737752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv" event={"ID":"8eb9a42f-a61f-4dcd-a209-eefe5dc24903","Type":"ContainerStarted","Data":"991a18d6ea30252769a0255b684088a11c518a37fcb3a38dd75ba314f78d5224"}
Jul 20 09:57:48.665688 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:48.665656    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:48.859545 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:48.859510    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:49.251854 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:49.251725    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:49.461689 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:49.461656    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:49.734603 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:49.734571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:49.734775 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:49.734635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/540eee1e-9d83-40b2-afb6-970f4df13724-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:49.734775 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:49.734730    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:49.734845 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:57:49.734795    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls podName:540eee1e-9d83-40b2-afb6-970f4df13724 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:05.734780346 +0000 UTC m=+97.783643971 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h8tvt" (UID: "540eee1e-9d83-40b2-afb6-970f4df13724") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:49.735075 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:49.735053    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/540eee1e-9d83-40b2-afb6-970f4df13724-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:57:50.060178 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:50.060075    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:50.457538 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:50.457501    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:50.940702 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:50.940610    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:51.257247 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:51.257212    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:51.458611 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:51.458569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:51.667713 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:51.667601    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:54.892054 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:54.892007    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:54.992614 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:54.992585    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:55.045183 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:55.045123    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:55.223638 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:55.223600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:55.370828 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:55.370796    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:55.470538 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:55.470501    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:55.798703 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:55.798672    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:55.874578 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:55.874547    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:55.980660 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:55.980627    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:56.194840 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:56.194759    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:56.270835 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:56.270797    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:56.590429 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:56.590397    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:56.799529 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:56.799491    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:57.194782 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:57.194742    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:58.005307 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:58.005261    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:58.106515 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:57:58.106475    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:00.093988 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.093923    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7jrnv" podStartSLOduration=29.50067768 podStartE2EDuration="31.093903785s" totalImagesPullingTime="1.593226105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:47.7578977 +0000 UTC m=+79.806761347" watchObservedRunningTime="2026-07-20 09:58:00.093903785 +0000 UTC m=+92.142767439"
Jul 20 09:58:00.094998 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.094978    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"]
Jul 20 09:58:00.097657 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.097643    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.101100 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.101081    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 20 09:58:00.101267 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.101157    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 20 09:58:00.101718 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.101702    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 20 09:58:00.101922 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.101905    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 20 09:58:00.101980 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.101936    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 20 09:58:00.102015 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.101996    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 20 09:58:00.126272 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.126246    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 20 09:58:00.134969 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.134942    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"]
Jul 20 09:58:00.219234 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.219196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6d07d18e-8a75-4128-b074-b8332c9cde4f-ca\") pod \"cluster-proxy-proxy-agent-6f55bfd5bd-vjrst\" (UID: \"6d07d18e-8a75-4128-b074-b8332c9cde4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.219234 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.219239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6d07d18e-8a75-4128-b074-b8332c9cde4f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f55bfd5bd-vjrst\" (UID: \"6d07d18e-8a75-4128-b074-b8332c9cde4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.219471 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.219270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bml7\" (UniqueName: \"kubernetes.io/projected/6d07d18e-8a75-4128-b074-b8332c9cde4f-kube-api-access-6bml7\") pod \"cluster-proxy-proxy-agent-6f55bfd5bd-vjrst\" (UID: \"6d07d18e-8a75-4128-b074-b8332c9cde4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.219471 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.219294    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6d07d18e-8a75-4128-b074-b8332c9cde4f-hub\") pod \"cluster-proxy-proxy-agent-6f55bfd5bd-vjrst\" (UID: \"6d07d18e-8a75-4128-b074-b8332c9cde4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.219471 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.219413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d07d18e-8a75-4128-b074-b8332c9cde4f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f55bfd5bd-vjrst\" (UID: \"6d07d18e-8a75-4128-b074-b8332c9cde4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.219471 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.219454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d07d18e-8a75-4128-b074-b8332c9cde4f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f55bfd5bd-vjrst\" (UID: \"6d07d18e-8a75-4128-b074-b8332c9cde4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.285789 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.285760    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:00.319935 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.319902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6d07d18e-8a75-4128-b074-b8332c9cde4f-ca\") pod \"cluster-proxy-proxy-agent-6f55bfd5bd-vjrst\" (UID: \"6d07d18e-8a75-4128-b074-b8332c9cde4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.319935 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.319942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6d07d18e-8a75-4128-b074-b8332c9cde4f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f55bfd5bd-vjrst\" (UID: \"6d07d18e-8a75-4128-b074-b8332c9cde4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.320159 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.319960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bml7\" (UniqueName: \"kubernetes.io/projected/6d07d18e-8a75-4128-b074-b8332c9cde4f-kube-api-access-6bml7\") pod \"cluster-proxy-proxy-agent-6f55bfd5bd-vjrst\" (UID: \"6d07d18e-8a75-4128-b074-b8332c9cde4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.320159 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.319977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6d07d18e-8a75-4128-b074-b8332c9cde4f-hub\") pod \"cluster-proxy-proxy-agent-6f55bfd5bd-vjrst\" (UID: \"6d07d18e-8a75-4128-b074-b8332c9cde4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.320285 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.320222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d07d18e-8a75-4128-b074-b8332c9cde4f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f55bfd5bd-vjrst\" (UID: \"6d07d18e-8a75-4128-b074-b8332c9cde4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.320339 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.320314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d07d18e-8a75-4128-b074-b8332c9cde4f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f55bfd5bd-vjrst\" (UID: \"6d07d18e-8a75-4128-b074-b8332c9cde4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.321748 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.321713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d07d18e-8a75-4128-b074-b8332c9cde4f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f55bfd5bd-vjrst\" (UID: \"6d07d18e-8a75-4128-b074-b8332c9cde4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.322619 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.322597    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6d07d18e-8a75-4128-b074-b8332c9cde4f-hub\") pod \"cluster-proxy-proxy-agent-6f55bfd5bd-vjrst\" (UID: \"6d07d18e-8a75-4128-b074-b8332c9cde4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.322748 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.322674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6d07d18e-8a75-4128-b074-b8332c9cde4f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f55bfd5bd-vjrst\" (UID: \"6d07d18e-8a75-4128-b074-b8332c9cde4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.322805 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.322703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6d07d18e-8a75-4128-b074-b8332c9cde4f-ca\") pod \"cluster-proxy-proxy-agent-6f55bfd5bd-vjrst\" (UID: \"6d07d18e-8a75-4128-b074-b8332c9cde4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.322913 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.322899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d07d18e-8a75-4128-b074-b8332c9cde4f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f55bfd5bd-vjrst\" (UID: \"6d07d18e-8a75-4128-b074-b8332c9cde4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.327733 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.327712    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bml7\" (UniqueName: \"kubernetes.io/projected/6d07d18e-8a75-4128-b074-b8332c9cde4f-kube-api-access-6bml7\") pod \"cluster-proxy-proxy-agent-6f55bfd5bd-vjrst\" (UID: \"6d07d18e-8a75-4128-b074-b8332c9cde4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.391277 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.391156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:00.415768 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.415739    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"
Jul 20 09:58:00.517682 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.517641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:00.577409 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.577378    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst"]
Jul 20 09:58:00.581012 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:58:00.580984    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d07d18e_8a75_4128_b074_b8332c9cde4f.slice/crio-349ae228dded6fbaff2f379a6cb7d19938ef7f9027b1995388faa9c74a042371 WatchSource:0}: Error finding container 349ae228dded6fbaff2f379a6cb7d19938ef7f9027b1995388faa9c74a042371: Status 404 returned error can't find the container with id 349ae228dded6fbaff2f379a6cb7d19938ef7f9027b1995388faa9c74a042371
Jul 20 09:58:00.644758 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.644669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:00.775338 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:00.775308    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst" event={"ID":"6d07d18e-8a75-4128-b074-b8332c9cde4f","Type":"ContainerStarted","Data":"349ae228dded6fbaff2f379a6cb7d19938ef7f9027b1995388faa9c74a042371"}
Jul 20 09:58:01.431213 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:01.431162    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rd75s\" (UID: \"4c1f99dc-12d6-4019-ae66-0d06db01e2a2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"
Jul 20 09:58:01.434497 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:01.434444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c1f99dc-12d6-4019-ae66-0d06db01e2a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rd75s\" (UID: \"4c1f99dc-12d6-4019-ae66-0d06db01e2a2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"
Jul 20 09:58:01.732639 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:01.732594    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"
Jul 20 09:58:02.560727 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:02.560686    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s"]
Jul 20 09:58:02.566103 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:58:02.566074    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1f99dc_12d6_4019_ae66_0d06db01e2a2.slice/crio-6f4a7338af15c2c58579345f7c43a98b532ce33db48dd42bf9c96baeac230f3f WatchSource:0}: Error finding container 6f4a7338af15c2c58579345f7c43a98b532ce33db48dd42bf9c96baeac230f3f: Status 404 returned error can't find the container with id 6f4a7338af15c2c58579345f7c43a98b532ce33db48dd42bf9c96baeac230f3f
Jul 20 09:58:02.784477 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:02.784437    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s" event={"ID":"4c1f99dc-12d6-4019-ae66-0d06db01e2a2","Type":"ContainerStarted","Data":"6f4a7338af15c2c58579345f7c43a98b532ce33db48dd42bf9c96baeac230f3f"}
Jul 20 09:58:02.896810 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:02.896773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:02.998523 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:02.998481    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:03.098575 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:03.098483    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:04.010489 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:04.010401    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:04.110503 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:04.110465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:04.180950 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:04.180891    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:04.793250 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:04.793211    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst" event={"ID":"6d07d18e-8a75-4128-b074-b8332c9cde4f","Type":"ContainerStarted","Data":"c33e8615dca2b6435c81fe7507c2e6a183d22316f4cbbeda8e5bb83c4d55bcc9"}
Jul 20 09:58:04.793436 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:04.793268    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst" event={"ID":"6d07d18e-8a75-4128-b074-b8332c9cde4f","Type":"ContainerStarted","Data":"50bcf355d07cfb7f6c933899ff2558a2423133d75e88aa23c33fc5acec501135"}
Jul 20 09:58:04.793436 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:04.793287    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst" event={"ID":"6d07d18e-8a75-4128-b074-b8332c9cde4f","Type":"ContainerStarted","Data":"4a83610d53c1ced457d2ee6b6af9d5a980e4febeb62cbfd8cad8b2bef578c948"}
Jul 20 09:58:04.818545 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:04.818486    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f55bfd5bd-vjrst" podStartSLOduration=1.6464565599999998 podStartE2EDuration="4.818463966s" totalImagesPullingTime="3.172007406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:04.818077059 +0000 UTC m=+96.866940706" watchObservedRunningTime="2026-07-20 09:58:04.818463966 +0000 UTC m=+96.867327614"
Jul 20 09:58:05.711736 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:05.711700    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:05.771105 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:05.771070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:58:05.773744 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:05.773716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/540eee1e-9d83-40b2-afb6-970f4df13724-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h8tvt\" (UID: \"540eee1e-9d83-40b2-afb6-970f4df13724\") " pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:58:05.802849 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:05.802813    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s" event={"ID":"4c1f99dc-12d6-4019-ae66-0d06db01e2a2","Type":"ContainerStarted","Data":"5b58a9a9513ee94dff43ddea9a9746d635cc01deb461a63c41027440c5723966"}
Jul 20 09:58:05.810743 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:05.810705    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:05.825048 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:05.824982    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rd75s" podStartSLOduration=34.053326845 podStartE2EDuration="36.824962228s" totalImagesPullingTime="2.771635383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:05.824053291 +0000 UTC m=+97.872916939" watchObservedRunningTime="2026-07-20 09:58:05.824962228 +0000 UTC m=+97.873825878"
Jul 20 09:58:05.941100 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:05.941068    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:05.995712 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:05.995627    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h8tvt"
Jul 20 09:58:06.013844 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:06.013802    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:06.144648 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:06.144616    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h8tvt"]
Jul 20 09:58:06.147841 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:58:06.147815    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540eee1e_9d83_40b2_afb6_970f4df13724.slice/crio-58d83339aa4d4ab4176fd0bb7792ac41c6efacae5fa435515d173feaaf26509f WatchSource:0}: Error finding container 58d83339aa4d4ab4176fd0bb7792ac41c6efacae5fa435515d173feaaf26509f: Status 404 returned error can't find the container with id 58d83339aa4d4ab4176fd0bb7792ac41c6efacae5fa435515d173feaaf26509f
Jul 20 09:58:06.594973 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:06.594905    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:06.654846 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:06.654796    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:06.815443 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:06.814917    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h8tvt" event={"ID":"540eee1e-9d83-40b2-afb6-970f4df13724","Type":"ContainerStarted","Data":"58d83339aa4d4ab4176fd0bb7792ac41c6efacae5fa435515d173feaaf26509f"}
Jul 20 09:58:07.695451 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:07.695413    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-r6xpf"
Jul 20 09:58:07.792474 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:07.792391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:58:07.795339 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:07.795305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls\") pod \"image-registry-66778bc5c4-fn2ml\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") " pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:58:07.893299 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:07.893266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:58:07.896237 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:07.896208    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ffa1-8dfa-470e-be51-643f5adbc99c-metrics-tls\") pod \"dns-default-2hnhp\" (UID: \"8215ffa1-8dfa-470e-be51-643f5adbc99c\") " pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:58:08.098160 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:08.098071    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x8ldn\""
Jul 20 09:58:08.106233 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:08.106192    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:58:08.147719 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:08.147695    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g9txk\""
Jul 20 09:58:08.155663 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:08.155615    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:58:08.287957 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:08.287924    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66778bc5c4-fn2ml"]
Jul 20 09:58:08.291297 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:58:08.291266    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd510a0_1672_4049_9fb2_561f5aa8ea24.slice/crio-2ab4353624829e5c83f1f094fbb97fca9a2dd42c52c5286bd930d6c951a87df6 WatchSource:0}: Error finding container 2ab4353624829e5c83f1f094fbb97fca9a2dd42c52c5286bd930d6c951a87df6: Status 404 returned error can't find the container with id 2ab4353624829e5c83f1f094fbb97fca9a2dd42c52c5286bd930d6c951a87df6
Jul 20 09:58:08.338808 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:08.338647    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2hnhp"]
Jul 20 09:58:08.342061 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:58:08.342014    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8215ffa1_8dfa_470e_be51_643f5adbc99c.slice/crio-ce18433e0faedc82af14aa3fc12993549305cd6b636d48fc3fd2150f1d05ee41 WatchSource:0}: Error finding container ce18433e0faedc82af14aa3fc12993549305cd6b636d48fc3fd2150f1d05ee41: Status 404 returned error can't find the container with id ce18433e0faedc82af14aa3fc12993549305cd6b636d48fc3fd2150f1d05ee41
Jul 20 09:58:08.824634 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:08.824591    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2hnhp" event={"ID":"8215ffa1-8dfa-470e-be51-643f5adbc99c","Type":"ContainerStarted","Data":"ce18433e0faedc82af14aa3fc12993549305cd6b636d48fc3fd2150f1d05ee41"}
Jul 20 09:58:08.826379 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:08.826332    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml" event={"ID":"5cd510a0-1672-4049-9fb2-561f5aa8ea24","Type":"ContainerStarted","Data":"a02e2d94f4bd2bc2cb50d89621bed40919ecef7b7d8213c5e4bbc9e067d52587"}
Jul 20 09:58:08.826505 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:08.826402    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml" event={"ID":"5cd510a0-1672-4049-9fb2-561f5aa8ea24","Type":"ContainerStarted","Data":"2ab4353624829e5c83f1f094fbb97fca9a2dd42c52c5286bd930d6c951a87df6"}
Jul 20 09:58:08.827129 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:08.827105    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:58:11.267496 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:11.267457    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:11.360966 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:11.360912    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:12.018026 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:12.017981    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:12.119493 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:12.119397    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:12.223846 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:12.223813    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:12.842703 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:12.842665    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h8tvt" event={"ID":"540eee1e-9d83-40b2-afb6-970f4df13724","Type":"ContainerStarted","Data":"0d737594571dc9769803f52acaabf74fa5e539b2db5c7cfbd78e516f581f1dd4"}
Jul 20 09:58:12.844715 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:12.844687    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2hnhp" event={"ID":"8215ffa1-8dfa-470e-be51-643f5adbc99c","Type":"ContainerStarted","Data":"5b71b74862382fee381091bad7adc88238fcc84e0ad6dda2d5c425359aa00d76"}
Jul 20 09:58:12.844852 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:12.844799    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2hnhp" event={"ID":"8215ffa1-8dfa-470e-be51-643f5adbc99c","Type":"ContainerStarted","Data":"c37eefc54c6f542ffde28acbfd6a9fbe192081c689a7e54d2be0f558f49946f2"}
Jul 20 09:58:12.845093 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:12.845073    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:58:12.863010 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:12.862956    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml" podStartSLOduration=81.862936214 podStartE2EDuration="1m21.862936214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:08.851306363 +0000 UTC m=+100.900170011" watchObservedRunningTime="2026-07-20 09:58:12.862936214 +0000 UTC m=+104.911799864"
Jul 20 09:58:12.863185 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:12.863116    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2hnhp" podStartSLOduration=66.337887697 podStartE2EDuration="1m9.863108125s" totalImagesPullingTime="3.525220428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:12.861832493 +0000 UTC m=+104.910696140" watchObservedRunningTime="2026-07-20 09:58:12.863108125 +0000 UTC m=+104.911971772"
Jul 20 09:58:12.919398 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:12.919366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:12.994954 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:12.994856    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:13.602316 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:13.602280    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:14.411007 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:14.410977    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:14.507549 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:14.507509    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:15.859415 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:15.859373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h8tvt" event={"ID":"540eee1e-9d83-40b2-afb6-970f4df13724","Type":"ContainerStarted","Data":"d63195b683d4a38adfc5bcbe3f7cf1003cf87055aeec2c77dc5bd156748f22e2"}
Jul 20 09:58:16.965825 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:16.965795    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:17.041843 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:17.041797    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:18.447361 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:18.447327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:18.556902 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:18.556871    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:18.872003 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:18.871969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h8tvt" event={"ID":"540eee1e-9d83-40b2-afb6-970f4df13724","Type":"ContainerStarted","Data":"56c91a9292ad8532ed6a63f23e38d6c8fdd6e4c9e4babe06ca36dcf552cebd36"}
Jul 20 09:58:18.894528 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:18.894478    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-h8tvt" podStartSLOduration=33.901635398 podStartE2EDuration="45.894463993s" totalImagesPullingTime="11.992828595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:18.894200394 +0000 UTC m=+110.943064041" watchObservedRunningTime="2026-07-20 09:58:18.894463993 +0000 UTC m=+110.943327640"
Jul 20 09:58:22.853064 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:22.853032    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2hnhp"
Jul 20 09:58:23.932767 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:23.932733    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66778bc5c4-fn2ml"]
Jul 20 09:58:23.937160 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:23.937137    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:58:24.040868 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:24.040831    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:24.133832 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:24.133798    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:24.238618 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:24.238589    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:24.335756 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:24.335708    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:27.072671 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.072630    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:27.206667 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.206635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:27.316909 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.316876    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:27.362554 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.362471    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c4wwp"]
Jul 20 09:58:27.368312 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.368287    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.373588 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.373517    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 20 09:58:27.373588 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.373543    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2kv7c\""
Jul 20 09:58:27.373792 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.373589    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 20 09:58:27.373792 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.373601    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 20 09:58:27.375075 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.375055    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 20 09:58:27.408637 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.408600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:27.447010 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.446977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9fc14519-d15a-420e-a8af-06a2babcdc4b-node-exporter-textfile\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.447196 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.447030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kv9q\" (UniqueName: \"kubernetes.io/projected/9fc14519-d15a-420e-a8af-06a2babcdc4b-kube-api-access-2kv9q\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.447196 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.447091    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9fc14519-d15a-420e-a8af-06a2babcdc4b-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.447196 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.447130    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fc14519-d15a-420e-a8af-06a2babcdc4b-node-exporter-tls\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.447196 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.447154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fc14519-d15a-420e-a8af-06a2babcdc4b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.447334 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.447212    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fc14519-d15a-420e-a8af-06a2babcdc4b-metrics-client-ca\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.447334 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.447243    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9fc14519-d15a-420e-a8af-06a2babcdc4b-root\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.447334 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.447267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9fc14519-d15a-420e-a8af-06a2babcdc4b-node-exporter-wtmp\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.447424 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.447328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fc14519-d15a-420e-a8af-06a2babcdc4b-sys\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.548372 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.548336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9fc14519-d15a-420e-a8af-06a2babcdc4b-root\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.548551 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.548384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9fc14519-d15a-420e-a8af-06a2babcdc4b-node-exporter-wtmp\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.548551 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.548424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fc14519-d15a-420e-a8af-06a2babcdc4b-sys\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.548551 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.548447    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9fc14519-d15a-420e-a8af-06a2babcdc4b-root\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.548551 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.548464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9fc14519-d15a-420e-a8af-06a2babcdc4b-node-exporter-textfile\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.548551 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.548525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fc14519-d15a-420e-a8af-06a2babcdc4b-sys\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.548872 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.548600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9fc14519-d15a-420e-a8af-06a2babcdc4b-node-exporter-wtmp\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.548872 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.548617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kv9q\" (UniqueName: \"kubernetes.io/projected/9fc14519-d15a-420e-a8af-06a2babcdc4b-kube-api-access-2kv9q\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.548872 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.548660    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9fc14519-d15a-420e-a8af-06a2babcdc4b-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.548872 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.548685    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fc14519-d15a-420e-a8af-06a2babcdc4b-node-exporter-tls\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.548872 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.548709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fc14519-d15a-420e-a8af-06a2babcdc4b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.548872 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.548728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fc14519-d15a-420e-a8af-06a2babcdc4b-metrics-client-ca\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.548872 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.548803    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9fc14519-d15a-420e-a8af-06a2babcdc4b-node-exporter-textfile\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.548872 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:58:27.548815    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 20 09:58:27.548872 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:58:27.548874    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fc14519-d15a-420e-a8af-06a2babcdc4b-node-exporter-tls podName:9fc14519-d15a-420e-a8af-06a2babcdc4b nodeName:}" failed. No retries permitted until 2026-07-20 09:58:28.048852277 +0000 UTC m=+120.097715905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9fc14519-d15a-420e-a8af-06a2babcdc4b-node-exporter-tls") pod "node-exporter-c4wwp" (UID: "9fc14519-d15a-420e-a8af-06a2babcdc4b") : secret "node-exporter-tls" not found
Jul 20 09:58:27.549370 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.549345    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9fc14519-d15a-420e-a8af-06a2babcdc4b-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.549470 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.549349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fc14519-d15a-420e-a8af-06a2babcdc4b-metrics-client-ca\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.551232 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.551215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fc14519-d15a-420e-a8af-06a2babcdc4b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.561228 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.561206    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kv9q\" (UniqueName: \"kubernetes.io/projected/9fc14519-d15a-420e-a8af-06a2babcdc4b-kube-api-access-2kv9q\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:27.603281 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.603245    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:27.710657 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.710605    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:27.816629 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.816593    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:27.916574 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:27.916541    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:28.020427 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:28.020338    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:28.053666 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:28.053637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fc14519-d15a-420e-a8af-06a2babcdc4b-node-exporter-tls\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:28.056220 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:28.056193    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fc14519-d15a-420e-a8af-06a2babcdc4b-node-exporter-tls\") pod \"node-exporter-c4wwp\" (UID: \"9fc14519-d15a-420e-a8af-06a2babcdc4b\") " pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:28.120304 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:28.120253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:28.283287 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:28.283211    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2kv7c\""
Jul 20 09:58:28.291586 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:28.291563    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c4wwp"
Jul 20 09:58:28.303940 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:58:28.303914    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc14519_d15a_420e_a8af_06a2babcdc4b.slice/crio-09e6ef0b4c4917eef358772384f455a42f34712ded944dc97f43edc7a52fefc8 WatchSource:0}: Error finding container 09e6ef0b4c4917eef358772384f455a42f34712ded944dc97f43edc7a52fefc8: Status 404 returned error can't find the container with id 09e6ef0b4c4917eef358772384f455a42f34712ded944dc97f43edc7a52fefc8
Jul 20 09:58:28.334183 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:28.334123    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:28.471333 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:28.471306    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:28.632007 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:28.631909    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:28.906323 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:28.906238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4wwp" event={"ID":"9fc14519-d15a-420e-a8af-06a2babcdc4b","Type":"ContainerStarted","Data":"09e6ef0b4c4917eef358772384f455a42f34712ded944dc97f43edc7a52fefc8"}
Jul 20 09:58:29.911330 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:29.911295    2580 generic.go:356] "Generic (PLEG): container finished" podID="9fc14519-d15a-420e-a8af-06a2babcdc4b" containerID="b514682038ded56812a1300a85c6598700eacd718b1f6f62272000dc6eb84ec8" exitCode=0
Jul 20 09:58:29.911705 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:29.911369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4wwp" event={"ID":"9fc14519-d15a-420e-a8af-06a2babcdc4b","Type":"ContainerDied","Data":"b514682038ded56812a1300a85c6598700eacd718b1f6f62272000dc6eb84ec8"}
Jul 20 09:58:30.292525 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:30.292493    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:30.365493 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:30.365462    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:30.452938 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:30.452905    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:30.549653 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:30.549563    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:30.625037 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:30.625002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:30.725430 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:30.725399    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:30.828398 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:30.828313    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:30.916839 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:30.916806    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4wwp" event={"ID":"9fc14519-d15a-420e-a8af-06a2babcdc4b","Type":"ContainerStarted","Data":"437e0301cce5c2f4eee679c356b0c5689ce21859ac7916e121703d03dccc36f7"}
Jul 20 09:58:30.917246 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:30.916862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4wwp" event={"ID":"9fc14519-d15a-420e-a8af-06a2babcdc4b","Type":"ContainerStarted","Data":"b580fbc68eee240ed57f9aa33e849d4593f20a7afd8365ca2c750ab6d92902c0"}
Jul 20 09:58:30.934990 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:30.934943    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c4wwp" podStartSLOduration=2.949092845 podStartE2EDuration="3.934926228s" totalImagesPullingTime="913.362453ms" totalInitContainerRuntime="72.47093ms" isStatefulPod=true podCreationTimestamp="2026-07-20 09:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:30.934875455 +0000 UTC m=+122.983739103" watchObservedRunningTime="2026-07-20 09:58:30.934926228 +0000 UTC m=+122.983789875"
Jul 20 09:58:31.994722 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:31.994691    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:32.095511 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:32.095473    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:32.194888 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:32.194857    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:32.361027 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:32.360936    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:33.242956 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:33.242916    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:33.357574 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:33.357541    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:33.454346 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:33.454315    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:33.683247 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:33.683151    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:38.111430 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:38.111381    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:38.139535 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:38.139506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs\") pod \"network-metrics-daemon-jms2f\" (UID: \"5e4c9043-3e1d-49f1-8550-ce507005bcc4\") " pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:58:38.141983 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:38.141964    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4c9043-3e1d-49f1-8550-ce507005bcc4-metrics-certs\") pod \"network-metrics-daemon-jms2f\" (UID: \"5e4c9043-3e1d-49f1-8550-ce507005bcc4\") " pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:58:38.228116 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:38.228078    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:38.312294 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:38.312256    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:38.437678 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:38.437613    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9x8bv\""
Jul 20 09:58:38.444351 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:38.444330    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jms2f"
Jul 20 09:58:38.608455 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:38.608425    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jms2f"]
Jul 20 09:58:38.611398 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:58:38.611369    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4c9043_3e1d_49f1_8550_ce507005bcc4.slice/crio-6c3883b007c8bfcc7d0e2883f6667595b1262b9eef4326e36c75e204c9aec724 WatchSource:0}: Error finding container 6c3883b007c8bfcc7d0e2883f6667595b1262b9eef4326e36c75e204c9aec724: Status 404 returned error can't find the container with id 6c3883b007c8bfcc7d0e2883f6667595b1262b9eef4326e36c75e204c9aec724
Jul 20 09:58:38.945786 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:38.945755    2580 generic.go:356] "Generic (PLEG): container finished" podID="1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6" containerID="1e97b38d76b5d915488a7d9088b9852f3f776f26262b1d1b3a63b931fa8435f7" exitCode=0
Jul 20 09:58:38.946042 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:38.945833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn" event={"ID":"1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6","Type":"ContainerDied","Data":"1e97b38d76b5d915488a7d9088b9852f3f776f26262b1d1b3a63b931fa8435f7"}
Jul 20 09:58:38.946278 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:38.946247    2580 scope.go:117] "RemoveContainer" containerID="1e97b38d76b5d915488a7d9088b9852f3f776f26262b1d1b3a63b931fa8435f7"
Jul 20 09:58:38.947065 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:38.947045    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jms2f" event={"ID":"5e4c9043-3e1d-49f1-8550-ce507005bcc4","Type":"ContainerStarted","Data":"6c3883b007c8bfcc7d0e2883f6667595b1262b9eef4326e36c75e204c9aec724"}
Jul 20 09:58:39.953045 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:39.953012    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5npqn" event={"ID":"1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6","Type":"ContainerStarted","Data":"e615f95abd5b40f22e1557b986c16fc1e0958cbfa07cefe3f7028f52f4a83b0a"}
Jul 20 09:58:39.983604 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:39.983570    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:40.136939 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:40.136888    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:40.958425 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:40.958389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jms2f" event={"ID":"5e4c9043-3e1d-49f1-8550-ce507005bcc4","Type":"ContainerStarted","Data":"b1e4b89de00a6a8da3dd1760ff622d2604a4296e04d956cc84f258613ff69fae"}
Jul 20 09:58:40.958902 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:40.958440    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jms2f" event={"ID":"5e4c9043-3e1d-49f1-8550-ce507005bcc4","Type":"ContainerStarted","Data":"83c152b4c6065ad04fa4ed27cce86370421e8ec34f70ac9b99da367c1b91b47c"}
Jul 20 09:58:41.133515 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:41.133483    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:41.246489 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:41.246399    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:41.348008 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:41.347971    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:42.561700 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:42.561665    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:42.660418 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:42.660377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:44.335136 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:44.335076    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:44.441214 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:44.441138    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:48.953307 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:48.953243    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml" podUID="5cd510a0-1672-4049-9fb2-561f5aa8ea24" containerName="registry" containerID="cri-o://a02e2d94f4bd2bc2cb50d89621bed40919ecef7b7d8213c5e4bbc9e067d52587" gracePeriod=30
Jul 20 09:58:49.739421 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.739389    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:58:49.764800 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.764743    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jms2f" podStartSLOduration=140.344256077 podStartE2EDuration="2m21.764725536s" totalImagesPullingTime="1.420469459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:40.995205716 +0000 UTC m=+133.044069365" watchObservedRunningTime="2026-07-20 09:58:49.764725536 +0000 UTC m=+141.813589184"
Jul 20 09:58:49.835628 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.835594    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm6dk\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-kube-api-access-xm6dk\") pod \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") "
Jul 20 09:58:49.835805 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.835656    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-certificates\") pod \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") "
Jul 20 09:58:49.835805 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.835680    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5cd510a0-1672-4049-9fb2-561f5aa8ea24-trusted-ca\") pod \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") "
Jul 20 09:58:49.835805 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.835700    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls\") pod \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") "
Jul 20 09:58:49.835956 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.835928    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-bound-sa-token\") pod \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") "
Jul 20 09:58:49.836015 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.835993    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5cd510a0-1672-4049-9fb2-561f5aa8ea24-image-registry-private-configuration\") pod \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") "
Jul 20 09:58:49.836069 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.836056    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5cd510a0-1672-4049-9fb2-561f5aa8ea24-installation-pull-secrets\") pod \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") "
Jul 20 09:58:49.836125 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.836089    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5cd510a0-1672-4049-9fb2-561f5aa8ea24-ca-trust-extracted\") pod \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\" (UID: \"5cd510a0-1672-4049-9fb2-561f5aa8ea24\") "
Jul 20 09:58:49.836258 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.836230    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5cd510a0-1672-4049-9fb2-561f5aa8ea24" (UID: "5cd510a0-1672-4049-9fb2-561f5aa8ea24"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:58:49.836340 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.836248    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd510a0-1672-4049-9fb2-561f5aa8ea24-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5cd510a0-1672-4049-9fb2-561f5aa8ea24" (UID: "5cd510a0-1672-4049-9fb2-561f5aa8ea24"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:58:49.836461 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.836441    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-certificates\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 09:58:49.836530 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.836465    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5cd510a0-1672-4049-9fb2-561f5aa8ea24-trusted-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 09:58:49.838575 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.838547    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5cd510a0-1672-4049-9fb2-561f5aa8ea24" (UID: "5cd510a0-1672-4049-9fb2-561f5aa8ea24"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:58:49.838696 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.838611    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd510a0-1672-4049-9fb2-561f5aa8ea24-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5cd510a0-1672-4049-9fb2-561f5aa8ea24" (UID: "5cd510a0-1672-4049-9fb2-561f5aa8ea24"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:58:49.838828 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.838800    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-kube-api-access-xm6dk" (OuterVolumeSpecName: "kube-api-access-xm6dk") pod "5cd510a0-1672-4049-9fb2-561f5aa8ea24" (UID: "5cd510a0-1672-4049-9fb2-561f5aa8ea24"). InnerVolumeSpecName "kube-api-access-xm6dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:58:49.838828 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.838815    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd510a0-1672-4049-9fb2-561f5aa8ea24-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5cd510a0-1672-4049-9fb2-561f5aa8ea24" (UID: "5cd510a0-1672-4049-9fb2-561f5aa8ea24"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:58:49.838828 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.838797    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5cd510a0-1672-4049-9fb2-561f5aa8ea24" (UID: "5cd510a0-1672-4049-9fb2-561f5aa8ea24"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:58:49.845653 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.845629    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd510a0-1672-4049-9fb2-561f5aa8ea24-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5cd510a0-1672-4049-9fb2-561f5aa8ea24" (UID: "5cd510a0-1672-4049-9fb2-561f5aa8ea24"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 09:58:49.937563 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.937475    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5cd510a0-1672-4049-9fb2-561f5aa8ea24-installation-pull-secrets\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 09:58:49.937563 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.937511    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5cd510a0-1672-4049-9fb2-561f5aa8ea24-ca-trust-extracted\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 09:58:49.937563 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.937520    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm6dk\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-kube-api-access-xm6dk\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 09:58:49.937563 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.937533    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-registry-tls\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 09:58:49.937563 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.937542    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5cd510a0-1672-4049-9fb2-561f5aa8ea24-bound-sa-token\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 09:58:49.937563 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.937551    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5cd510a0-1672-4049-9fb2-561f5aa8ea24-image-registry-private-configuration\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 09:58:49.989557 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.989522    2580 generic.go:356] "Generic (PLEG): container finished" podID="5cd510a0-1672-4049-9fb2-561f5aa8ea24" containerID="a02e2d94f4bd2bc2cb50d89621bed40919ecef7b7d8213c5e4bbc9e067d52587" exitCode=0
Jul 20 09:58:49.989557 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.989559    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml" event={"ID":"5cd510a0-1672-4049-9fb2-561f5aa8ea24","Type":"ContainerDied","Data":"a02e2d94f4bd2bc2cb50d89621bed40919ecef7b7d8213c5e4bbc9e067d52587"}
Jul 20 09:58:49.990075 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.989584    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml"
Jul 20 09:58:49.990075 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.989594    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66778bc5c4-fn2ml" event={"ID":"5cd510a0-1672-4049-9fb2-561f5aa8ea24","Type":"ContainerDied","Data":"2ab4353624829e5c83f1f094fbb97fca9a2dd42c52c5286bd930d6c951a87df6"}
Jul 20 09:58:49.990075 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:49.989614    2580 scope.go:117] "RemoveContainer" containerID="a02e2d94f4bd2bc2cb50d89621bed40919ecef7b7d8213c5e4bbc9e067d52587"
Jul 20 09:58:50.001229 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:50.001204    2580 scope.go:117] "RemoveContainer" containerID="a02e2d94f4bd2bc2cb50d89621bed40919ecef7b7d8213c5e4bbc9e067d52587"
Jul 20 09:58:50.001569 ip-10-0-133-160 kubenswrapper[2580]: E0720 09:58:50.001548    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02e2d94f4bd2bc2cb50d89621bed40919ecef7b7d8213c5e4bbc9e067d52587\": container with ID starting with a02e2d94f4bd2bc2cb50d89621bed40919ecef7b7d8213c5e4bbc9e067d52587 not found: ID does not exist" containerID="a02e2d94f4bd2bc2cb50d89621bed40919ecef7b7d8213c5e4bbc9e067d52587"
Jul 20 09:58:50.001626 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:50.001577    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02e2d94f4bd2bc2cb50d89621bed40919ecef7b7d8213c5e4bbc9e067d52587"} err="failed to get container status \"a02e2d94f4bd2bc2cb50d89621bed40919ecef7b7d8213c5e4bbc9e067d52587\": rpc error: code = NotFound desc = could not find container \"a02e2d94f4bd2bc2cb50d89621bed40919ecef7b7d8213c5e4bbc9e067d52587\": container with ID starting with a02e2d94f4bd2bc2cb50d89621bed40919ecef7b7d8213c5e4bbc9e067d52587 not found: ID does not exist"
Jul 20 09:58:50.011538 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:50.011489    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66778bc5c4-fn2ml"]
Jul 20 09:58:50.017763 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:50.017737    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66778bc5c4-fn2ml"]
Jul 20 09:58:50.410007 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:50.409964    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cd510a0-1672-4049-9fb2-561f5aa8ea24" path="/var/lib/kubelet/pods/5cd510a0-1672-4049-9fb2-561f5aa8ea24/volumes"
Jul 20 09:58:54.764466 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:54.764433    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:54.882913 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:54.882881    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:59.023650 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:59.023614    2580 generic.go:356] "Generic (PLEG): container finished" podID="944513c3-bdd1-40f5-8ffd-bd72d60e2d50" containerID="251bd64b4566680b672803757eabe4f4b2b82a68cb4d1c14224377f5fdcaa756" exitCode=0
Jul 20 09:58:59.024046 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:59.023661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-rm45p" event={"ID":"944513c3-bdd1-40f5-8ffd-bd72d60e2d50","Type":"ContainerDied","Data":"251bd64b4566680b672803757eabe4f4b2b82a68cb4d1c14224377f5fdcaa756"}
Jul 20 09:58:59.024046 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:58:59.023969    2580 scope.go:117] "RemoveContainer" containerID="251bd64b4566680b672803757eabe4f4b2b82a68cb4d1c14224377f5fdcaa756"
Jul 20 09:59:00.028648 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:00.028603    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-rm45p" event={"ID":"944513c3-bdd1-40f5-8ffd-bd72d60e2d50","Type":"ContainerStarted","Data":"c21d4dda91e7bce6a96ef3e7c52d385103b7c25bca3a6999fc1e1b03a00566d8"}
Jul 20 09:59:46.702911 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:46.702878    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:46.902312 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:46.902279    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:51.799537 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.799495    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-786b8f867b-bmpv5"]
Jul 20 09:59:51.799917 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.799742    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd510a0-1672-4049-9fb2-561f5aa8ea24" containerName="registry"
Jul 20 09:59:51.799917 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.799753    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd510a0-1672-4049-9fb2-561f5aa8ea24" containerName="registry"
Jul 20 09:59:51.799917 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.799807    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cd510a0-1672-4049-9fb2-561f5aa8ea24" containerName="registry"
Jul 20 09:59:51.802657 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.802641    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.804899 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.804868    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-44vkn\""
Jul 20 09:59:51.805073 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.805031    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 20 09:59:51.805288 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.805269    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 20 09:59:51.805730 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.805713    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 20 09:59:51.805965 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.805943    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 20 09:59:51.806047 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.805954    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 20 09:59:51.814666 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.814635    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 20 09:59:51.821308 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.821280    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-786b8f867b-bmpv5"]
Jul 20 09:59:51.890660 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.890629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bb35413a-c78d-4e16-8355-ae387bcb9260-secret-telemeter-client\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.890870 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.890667    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9zwm\" (UniqueName: \"kubernetes.io/projected/bb35413a-c78d-4e16-8355-ae387bcb9260-kube-api-access-b9zwm\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.890870 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.890688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb35413a-c78d-4e16-8355-ae387bcb9260-serving-certs-ca-bundle\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.890870 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.890752    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb35413a-c78d-4e16-8355-ae387bcb9260-metrics-client-ca\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.890870 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.890813    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bb35413a-c78d-4e16-8355-ae387bcb9260-telemeter-client-tls\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.890870 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.890840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb35413a-c78d-4e16-8355-ae387bcb9260-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.890870 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.890867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb35413a-c78d-4e16-8355-ae387bcb9260-telemeter-trusted-ca-bundle\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.891108 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.890915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bb35413a-c78d-4e16-8355-ae387bcb9260-federate-client-tls\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.991534 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.991498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bb35413a-c78d-4e16-8355-ae387bcb9260-secret-telemeter-client\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.991534 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.991538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9zwm\" (UniqueName: \"kubernetes.io/projected/bb35413a-c78d-4e16-8355-ae387bcb9260-kube-api-access-b9zwm\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.991766 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.991556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb35413a-c78d-4e16-8355-ae387bcb9260-serving-certs-ca-bundle\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.991766 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.991699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb35413a-c78d-4e16-8355-ae387bcb9260-metrics-client-ca\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.991766 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.991743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bb35413a-c78d-4e16-8355-ae387bcb9260-telemeter-client-tls\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.991766 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.991763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb35413a-c78d-4e16-8355-ae387bcb9260-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.991982 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.991792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb35413a-c78d-4e16-8355-ae387bcb9260-telemeter-trusted-ca-bundle\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.991982 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.991854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bb35413a-c78d-4e16-8355-ae387bcb9260-federate-client-tls\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.992401 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.992363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb35413a-c78d-4e16-8355-ae387bcb9260-metrics-client-ca\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.992514 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.992348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb35413a-c78d-4e16-8355-ae387bcb9260-serving-certs-ca-bundle\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.993082 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.993060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb35413a-c78d-4e16-8355-ae387bcb9260-telemeter-trusted-ca-bundle\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.994428 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.994400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb35413a-c78d-4e16-8355-ae387bcb9260-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.994537 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.994469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bb35413a-c78d-4e16-8355-ae387bcb9260-telemeter-client-tls\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.994728 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.994710    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bb35413a-c78d-4e16-8355-ae387bcb9260-federate-client-tls\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.994783 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.994744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bb35413a-c78d-4e16-8355-ae387bcb9260-secret-telemeter-client\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:51.999738 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:51.999716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9zwm\" (UniqueName: \"kubernetes.io/projected/bb35413a-c78d-4e16-8355-ae387bcb9260-kube-api-access-b9zwm\") pod \"telemeter-client-786b8f867b-bmpv5\" (UID: \"bb35413a-c78d-4e16-8355-ae387bcb9260\") " pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:52.112128 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:52.112033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5"
Jul 20 09:59:52.253415 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:52.253382    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-786b8f867b-bmpv5"]
Jul 20 09:59:52.257863 ip-10-0-133-160 kubenswrapper[2580]: W0720 09:59:52.257836    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb35413a_c78d_4e16_8355_ae387bcb9260.slice/crio-cd4ed12dfc957dcfda495d3e66c4a512927b90c561907c3a64e70ef160257824 WatchSource:0}: Error finding container cd4ed12dfc957dcfda495d3e66c4a512927b90c561907c3a64e70ef160257824: Status 404 returned error can't find the container with id cd4ed12dfc957dcfda495d3e66c4a512927b90c561907c3a64e70ef160257824
Jul 20 09:59:53.211564 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:53.211526    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5" event={"ID":"bb35413a-c78d-4e16-8355-ae387bcb9260","Type":"ContainerStarted","Data":"cd4ed12dfc957dcfda495d3e66c4a512927b90c561907c3a64e70ef160257824"}
Jul 20 09:59:54.217747 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:54.217717    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5" event={"ID":"bb35413a-c78d-4e16-8355-ae387bcb9260","Type":"ContainerStarted","Data":"e243e902907ac0bf0917fd9ef91bb0355f54622b82d6e27147f74de4ea85259a"}
Jul 20 09:59:55.222861 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:55.222826    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5" event={"ID":"bb35413a-c78d-4e16-8355-ae387bcb9260","Type":"ContainerStarted","Data":"a3210f27a49a84b7928972d400e6251c49fca80e76d5438a81158181ab77db72"}
Jul 20 09:59:55.223364 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:55.222894    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5" event={"ID":"bb35413a-c78d-4e16-8355-ae387bcb9260","Type":"ContainerStarted","Data":"efa7beee49f3ae5bb4251f52815f1ba2bf249139738a20650ff376c523f73e27"}
Jul 20 09:59:55.248531 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:55.248482    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-786b8f867b-bmpv5" podStartSLOduration=1.635561144 podStartE2EDuration="4.248465742s" totalImagesPullingTime="2.612904598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:55.24689648 +0000 UTC m=+207.295760128" watchObservedRunningTime="2026-07-20 09:59:55.248465742 +0000 UTC m=+207.297329396"
Jul 20 09:59:56.154602 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:56.154561    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:56.254381 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:56.254340    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:56.360678 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:56.360641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:56.891368 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:56.891327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:56.993484 ip-10-0-133-160 kubenswrapper[2580]: I0720 09:59:56.993449    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:07.533765 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:00:07.533726    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:07.638354 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:00:07.638321    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:16.162086 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:00:16.162024    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:26.712522 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:00:26.712485    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:26.779505 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:00:26.779470    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:29.755415 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:00:29.755374    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:29.824998 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:00:29.824961    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:36.804204 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:00:36.804143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:01.983702 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:01:01.983590    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:02.083246 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:01:02.083215    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:06.760105 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:01:06.760068    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:06.861119 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:01:06.861091    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:13.907822 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:01:13.907784    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:13.969996 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:01:13.969963    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:17.795573 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:01:17.795541    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:17.898156 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:01:17.898117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:17.959078 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:01:17.959043    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:26.256122 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:01:26.256064    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:26.355522 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:01:26.355478    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:26.439060 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:01:26.439032    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:28.325632 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:01:28.325606    2580 kubelet.go:1648] "Image garbage collection succeeded"
Jul 20 10:02:52.618877 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:52.618841    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:02:54.000696 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.000661    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx"]
Jul 20 10:02:54.003825 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.003810    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx"
Jul 20 10:02:54.005839 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.005818    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kznnl\""
Jul 20 10:02:54.005961 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.005856    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 20 10:02:54.006327 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.006312    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 20 10:02:54.016503 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.016468    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx"]
Jul 20 10:02:54.029944 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.029919    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f091a726-e40d-4af5-bbaa-0a6d929233dc-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx\" (UID: \"f091a726-e40d-4af5-bbaa-0a6d929233dc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx"
Jul 20 10:02:54.030059 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.029947    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f091a726-e40d-4af5-bbaa-0a6d929233dc-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx\" (UID: \"f091a726-e40d-4af5-bbaa-0a6d929233dc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx"
Jul 20 10:02:54.030059 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.029967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kvm2\" (UniqueName: \"kubernetes.io/projected/f091a726-e40d-4af5-bbaa-0a6d929233dc-kube-api-access-5kvm2\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx\" (UID: \"f091a726-e40d-4af5-bbaa-0a6d929233dc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx"
Jul 20 10:02:54.034784 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.034761    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:02:54.130810 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.130776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kvm2\" (UniqueName: \"kubernetes.io/projected/f091a726-e40d-4af5-bbaa-0a6d929233dc-kube-api-access-5kvm2\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx\" (UID: \"f091a726-e40d-4af5-bbaa-0a6d929233dc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx"
Jul 20 10:02:54.130950 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.130869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f091a726-e40d-4af5-bbaa-0a6d929233dc-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx\" (UID: \"f091a726-e40d-4af5-bbaa-0a6d929233dc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx"
Jul 20 10:02:54.130950 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.130885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f091a726-e40d-4af5-bbaa-0a6d929233dc-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx\" (UID: \"f091a726-e40d-4af5-bbaa-0a6d929233dc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx"
Jul 20 10:02:54.131227 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.131211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f091a726-e40d-4af5-bbaa-0a6d929233dc-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx\" (UID: \"f091a726-e40d-4af5-bbaa-0a6d929233dc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx"
Jul 20 10:02:54.131293 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.131253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f091a726-e40d-4af5-bbaa-0a6d929233dc-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx\" (UID: \"f091a726-e40d-4af5-bbaa-0a6d929233dc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx"
Jul 20 10:02:54.139248 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.139228    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kvm2\" (UniqueName: \"kubernetes.io/projected/f091a726-e40d-4af5-bbaa-0a6d929233dc-kube-api-access-5kvm2\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx\" (UID: \"f091a726-e40d-4af5-bbaa-0a6d929233dc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx"
Jul 20 10:02:54.312936 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.312855    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx"
Jul 20 10:02:54.442832 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.442804    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx"]
Jul 20 10:02:54.446016 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:02:54.445983    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf091a726_e40d_4af5_bbaa_0a6d929233dc.slice/crio-35d4d67f3776f565607625726f8697cd3958fc42b6c9dfe6d548d9e65fd58df3 WatchSource:0}: Error finding container 35d4d67f3776f565607625726f8697cd3958fc42b6c9dfe6d548d9e65fd58df3: Status 404 returned error can't find the container with id 35d4d67f3776f565607625726f8697cd3958fc42b6c9dfe6d548d9e65fd58df3
Jul 20 10:02:54.448021 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.448004    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:02:54.837439 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:54.837410    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx" event={"ID":"f091a726-e40d-4af5-bbaa-0a6d929233dc","Type":"ContainerStarted","Data":"35d4d67f3776f565607625726f8697cd3958fc42b6c9dfe6d548d9e65fd58df3"}
Jul 20 10:02:56.060180 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:56.060099    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:02:56.112440 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:02:56.112400    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:00.861270 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:00.861160    2580 generic.go:356] "Generic (PLEG): container finished" podID="f091a726-e40d-4af5-bbaa-0a6d929233dc" containerID="505bc4375863a1011cadd59683bbde1ddc751ce268782223722d25eb0c0db43f" exitCode=0
Jul 20 10:03:00.861270 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:00.861200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx" event={"ID":"f091a726-e40d-4af5-bbaa-0a6d929233dc","Type":"ContainerDied","Data":"505bc4375863a1011cadd59683bbde1ddc751ce268782223722d25eb0c0db43f"}
Jul 20 10:03:02.870727 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:02.870692    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx" event={"ID":"f091a726-e40d-4af5-bbaa-0a6d929233dc","Type":"ContainerStarted","Data":"d3495114fcba9dc212c6918f624751fc7ded58bb67f96d395d103171f3c68fd9"}
Jul 20 10:03:03.875923 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:03.875891    2580 generic.go:356] "Generic (PLEG): container finished" podID="f091a726-e40d-4af5-bbaa-0a6d929233dc" containerID="d3495114fcba9dc212c6918f624751fc7ded58bb67f96d395d103171f3c68fd9" exitCode=0
Jul 20 10:03:03.876322 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:03.875939    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx" event={"ID":"f091a726-e40d-4af5-bbaa-0a6d929233dc","Type":"ContainerDied","Data":"d3495114fcba9dc212c6918f624751fc7ded58bb67f96d395d103171f3c68fd9"}
Jul 20 10:03:10.907815 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:10.907780    2580 generic.go:356] "Generic (PLEG): container finished" podID="f091a726-e40d-4af5-bbaa-0a6d929233dc" containerID="a39cdf337201e7d76c3e6d8d2da11de5bee00ded1712e75fd6d82d27050e4e0f" exitCode=0
Jul 20 10:03:10.908315 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:10.907825    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx" event={"ID":"f091a726-e40d-4af5-bbaa-0a6d929233dc","Type":"ContainerDied","Data":"a39cdf337201e7d76c3e6d8d2da11de5bee00ded1712e75fd6d82d27050e4e0f"}
Jul 20 10:03:12.034436 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:12.034413    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx"
Jul 20 10:03:12.070755 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:12.070728    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f091a726-e40d-4af5-bbaa-0a6d929233dc-util\") pod \"f091a726-e40d-4af5-bbaa-0a6d929233dc\" (UID: \"f091a726-e40d-4af5-bbaa-0a6d929233dc\") "
Jul 20 10:03:12.070911 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:12.070769    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kvm2\" (UniqueName: \"kubernetes.io/projected/f091a726-e40d-4af5-bbaa-0a6d929233dc-kube-api-access-5kvm2\") pod \"f091a726-e40d-4af5-bbaa-0a6d929233dc\" (UID: \"f091a726-e40d-4af5-bbaa-0a6d929233dc\") "
Jul 20 10:03:12.070911 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:12.070813    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f091a726-e40d-4af5-bbaa-0a6d929233dc-bundle\") pod \"f091a726-e40d-4af5-bbaa-0a6d929233dc\" (UID: \"f091a726-e40d-4af5-bbaa-0a6d929233dc\") "
Jul 20 10:03:12.071455 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:12.071432    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f091a726-e40d-4af5-bbaa-0a6d929233dc-bundle" (OuterVolumeSpecName: "bundle") pod "f091a726-e40d-4af5-bbaa-0a6d929233dc" (UID: "f091a726-e40d-4af5-bbaa-0a6d929233dc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:03:12.073089 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:12.073067    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f091a726-e40d-4af5-bbaa-0a6d929233dc-kube-api-access-5kvm2" (OuterVolumeSpecName: "kube-api-access-5kvm2") pod "f091a726-e40d-4af5-bbaa-0a6d929233dc" (UID: "f091a726-e40d-4af5-bbaa-0a6d929233dc"). InnerVolumeSpecName "kube-api-access-5kvm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:03:12.074793 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:12.074770    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f091a726-e40d-4af5-bbaa-0a6d929233dc-util" (OuterVolumeSpecName: "util") pod "f091a726-e40d-4af5-bbaa-0a6d929233dc" (UID: "f091a726-e40d-4af5-bbaa-0a6d929233dc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:03:12.172127 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:12.172035    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kvm2\" (UniqueName: \"kubernetes.io/projected/f091a726-e40d-4af5-bbaa-0a6d929233dc-kube-api-access-5kvm2\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:03:12.172127 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:12.172073    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f091a726-e40d-4af5-bbaa-0a6d929233dc-bundle\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:03:12.172127 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:12.172083    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f091a726-e40d-4af5-bbaa-0a6d929233dc-util\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:03:12.916871 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:12.916844    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx"
Jul 20 10:03:12.916871 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:12.916857    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s88lx" event={"ID":"f091a726-e40d-4af5-bbaa-0a6d929233dc","Type":"ContainerDied","Data":"35d4d67f3776f565607625726f8697cd3958fc42b6c9dfe6d548d9e65fd58df3"}
Jul 20 10:03:12.917118 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:12.916902    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35d4d67f3776f565607625726f8697cd3958fc42b6c9dfe6d548d9e65fd58df3"
Jul 20 10:03:15.230408 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:15.230377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:15.336750 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:15.336717    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:16.771705 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:16.771675    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc"]
Jul 20 10:03:16.772071 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:16.771925    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f091a726-e40d-4af5-bbaa-0a6d929233dc" containerName="pull"
Jul 20 10:03:16.772071 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:16.771935    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f091a726-e40d-4af5-bbaa-0a6d929233dc" containerName="pull"
Jul 20 10:03:16.772071 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:16.771953    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f091a726-e40d-4af5-bbaa-0a6d929233dc" containerName="util"
Jul 20 10:03:16.772071 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:16.771958    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f091a726-e40d-4af5-bbaa-0a6d929233dc" containerName="util"
Jul 20 10:03:16.772071 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:16.771965    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f091a726-e40d-4af5-bbaa-0a6d929233dc" containerName="extract"
Jul 20 10:03:16.772071 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:16.771971    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f091a726-e40d-4af5-bbaa-0a6d929233dc" containerName="extract"
Jul 20 10:03:16.772071 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:16.772013    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f091a726-e40d-4af5-bbaa-0a6d929233dc" containerName="extract"
Jul 20 10:03:16.773936 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:16.773920    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc"
Jul 20 10:03:16.776072 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:16.776051    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 20 10:03:16.777275 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:16.777259    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 20 10:03:16.777330 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:16.777302    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-dlcqn\""
Jul 20 10:03:16.777764 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:16.777747    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 20 10:03:16.806593 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:16.806562    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc"]
Jul 20 10:03:16.906452 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:16.906413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/378938f9-e222-4e0f-bbaa-383b0f72b650-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc\" (UID: \"378938f9-e222-4e0f-bbaa-383b0f72b650\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc"
Jul 20 10:03:16.906626 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:16.906476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9mbs\" (UniqueName: \"kubernetes.io/projected/378938f9-e222-4e0f-bbaa-383b0f72b650-kube-api-access-v9mbs\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc\" (UID: \"378938f9-e222-4e0f-bbaa-383b0f72b650\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc"
Jul 20 10:03:17.006803 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:17.006772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/378938f9-e222-4e0f-bbaa-383b0f72b650-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc\" (UID: \"378938f9-e222-4e0f-bbaa-383b0f72b650\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc"
Jul 20 10:03:17.006990 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:17.006835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9mbs\" (UniqueName: \"kubernetes.io/projected/378938f9-e222-4e0f-bbaa-383b0f72b650-kube-api-access-v9mbs\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc\" (UID: \"378938f9-e222-4e0f-bbaa-383b0f72b650\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc"
Jul 20 10:03:17.009630 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:17.009608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/378938f9-e222-4e0f-bbaa-383b0f72b650-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc\" (UID: \"378938f9-e222-4e0f-bbaa-383b0f72b650\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc"
Jul 20 10:03:17.022432 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:17.022371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9mbs\" (UniqueName: \"kubernetes.io/projected/378938f9-e222-4e0f-bbaa-383b0f72b650-kube-api-access-v9mbs\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc\" (UID: \"378938f9-e222-4e0f-bbaa-383b0f72b650\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc"
Jul 20 10:03:17.083538 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:17.083506    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc"
Jul 20 10:03:17.223520 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:17.223485    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc"]
Jul 20 10:03:17.228843 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:03:17.228815    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378938f9_e222_4e0f_bbaa_383b0f72b650.slice/crio-9d9cb0dc086d35315c2de549c13f031de655e5b19ac6ddfd2dae620d5ebd3823 WatchSource:0}: Error finding container 9d9cb0dc086d35315c2de549c13f031de655e5b19ac6ddfd2dae620d5ebd3823: Status 404 returned error can't find the container with id 9d9cb0dc086d35315c2de549c13f031de655e5b19ac6ddfd2dae620d5ebd3823
Jul 20 10:03:17.935900 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:17.935867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc" event={"ID":"378938f9-e222-4e0f-bbaa-383b0f72b650","Type":"ContainerStarted","Data":"9d9cb0dc086d35315c2de549c13f031de655e5b19ac6ddfd2dae620d5ebd3823"}
Jul 20 10:03:20.436599 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.436486    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:20.750983 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.750954    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:20.755304 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.755282    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-kd6w2"]
Jul 20 10:03:20.758581 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.758562    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-kd6w2"
Jul 20 10:03:20.761787 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.761771    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-m8g5k\""
Jul 20 10:03:20.761896 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.761771    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 20 10:03:20.762192 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.762153    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 20 10:03:20.775432 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.775408    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-kd6w2"]
Jul 20 10:03:20.835779 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.835750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/50b5b7c8-b816-444c-8e7e-5b7c16927ceb-cabundle0\") pod \"keda-operator-77f648d694-kd6w2\" (UID: \"50b5b7c8-b816-444c-8e7e-5b7c16927ceb\") " pod="openshift-keda/keda-operator-77f648d694-kd6w2"
Jul 20 10:03:20.835779 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.835783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/50b5b7c8-b816-444c-8e7e-5b7c16927ceb-certificates\") pod \"keda-operator-77f648d694-kd6w2\" (UID: \"50b5b7c8-b816-444c-8e7e-5b7c16927ceb\") " pod="openshift-keda/keda-operator-77f648d694-kd6w2"
Jul 20 10:03:20.835993 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.835811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpx97\" (UniqueName: \"kubernetes.io/projected/50b5b7c8-b816-444c-8e7e-5b7c16927ceb-kube-api-access-fpx97\") pod \"keda-operator-77f648d694-kd6w2\" (UID: \"50b5b7c8-b816-444c-8e7e-5b7c16927ceb\") " pod="openshift-keda/keda-operator-77f648d694-kd6w2"
Jul 20 10:03:20.851789 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.851737    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:20.936584 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.936541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/50b5b7c8-b816-444c-8e7e-5b7c16927ceb-cabundle0\") pod \"keda-operator-77f648d694-kd6w2\" (UID: \"50b5b7c8-b816-444c-8e7e-5b7c16927ceb\") " pod="openshift-keda/keda-operator-77f648d694-kd6w2"
Jul 20 10:03:20.936788 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.936588    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/50b5b7c8-b816-444c-8e7e-5b7c16927ceb-certificates\") pod \"keda-operator-77f648d694-kd6w2\" (UID: \"50b5b7c8-b816-444c-8e7e-5b7c16927ceb\") " pod="openshift-keda/keda-operator-77f648d694-kd6w2"
Jul 20 10:03:20.936788 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.936631    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpx97\" (UniqueName: \"kubernetes.io/projected/50b5b7c8-b816-444c-8e7e-5b7c16927ceb-kube-api-access-fpx97\") pod \"keda-operator-77f648d694-kd6w2\" (UID: \"50b5b7c8-b816-444c-8e7e-5b7c16927ceb\") " pod="openshift-keda/keda-operator-77f648d694-kd6w2"
Jul 20 10:03:20.936895 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:03:20.936784    2580 secret.go:281] references non-existent secret key: ca.crt
Jul 20 10:03:20.936895 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:03:20.936809    2580 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 20 10:03:20.936895 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:03:20.936820    2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-kd6w2: references non-existent secret key: ca.crt
Jul 20 10:03:20.936895 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:03:20.936881    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50b5b7c8-b816-444c-8e7e-5b7c16927ceb-certificates podName:50b5b7c8-b816-444c-8e7e-5b7c16927ceb nodeName:}" failed. No retries permitted until 2026-07-20 10:03:21.436860044 +0000 UTC m=+413.485723672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/50b5b7c8-b816-444c-8e7e-5b7c16927ceb-certificates") pod "keda-operator-77f648d694-kd6w2" (UID: "50b5b7c8-b816-444c-8e7e-5b7c16927ceb") : references non-existent secret key: ca.crt
Jul 20 10:03:20.937245 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.937227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/50b5b7c8-b816-444c-8e7e-5b7c16927ceb-cabundle0\") pod \"keda-operator-77f648d694-kd6w2\" (UID: \"50b5b7c8-b816-444c-8e7e-5b7c16927ceb\") " pod="openshift-keda/keda-operator-77f648d694-kd6w2"
Jul 20 10:03:20.944730 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.944704    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpx97\" (UniqueName: \"kubernetes.io/projected/50b5b7c8-b816-444c-8e7e-5b7c16927ceb-kube-api-access-fpx97\") pod \"keda-operator-77f648d694-kd6w2\" (UID: \"50b5b7c8-b816-444c-8e7e-5b7c16927ceb\") " pod="openshift-keda/keda-operator-77f648d694-kd6w2"
Jul 20 10:03:20.949111 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.949082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc" event={"ID":"378938f9-e222-4e0f-bbaa-383b0f72b650","Type":"ContainerStarted","Data":"0cf3f43147b3d4eca99fc9677745703243e9111bc13621396665420f4dc9a7f8"}
Jul 20 10:03:20.949242 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.949210    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc"
Jul 20 10:03:20.959597 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.959571    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:20.970919 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:20.970864    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc" podStartSLOduration=2.181756283 podStartE2EDuration="4.970845774s" totalImagesPullingTime="2.789089491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:03:20.9704572 +0000 UTC m=+413.019320847" watchObservedRunningTime="2026-07-20 10:03:20.970845774 +0000 UTC m=+413.019709422"
Jul 20 10:03:21.255096 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.255059    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:21.381154 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.381118    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:21.394662 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.394640    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-v9gzk"]
Jul 20 10:03:21.397784 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.397769    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-v9gzk"
Jul 20 10:03:21.400037 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.400011    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 20 10:03:21.408983 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.408958    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-v9gzk"]
Jul 20 10:03:21.440461 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.440425    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr79m\" (UniqueName: \"kubernetes.io/projected/f0c8d182-15f8-4eb1-90c5-0e775deffc23-kube-api-access-qr79m\") pod \"keda-admission-56f9798bb4-v9gzk\" (UID: \"f0c8d182-15f8-4eb1-90c5-0e775deffc23\") " pod="openshift-keda/keda-admission-56f9798bb4-v9gzk"
Jul 20 10:03:21.440800 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.440479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f0c8d182-15f8-4eb1-90c5-0e775deffc23-certificates\") pod \"keda-admission-56f9798bb4-v9gzk\" (UID: \"f0c8d182-15f8-4eb1-90c5-0e775deffc23\") " pod="openshift-keda/keda-admission-56f9798bb4-v9gzk"
Jul 20 10:03:21.440800 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.440550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/50b5b7c8-b816-444c-8e7e-5b7c16927ceb-certificates\") pod \"keda-operator-77f648d694-kd6w2\" (UID: \"50b5b7c8-b816-444c-8e7e-5b7c16927ceb\") " pod="openshift-keda/keda-operator-77f648d694-kd6w2"
Jul 20 10:03:21.443028 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.442996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/50b5b7c8-b816-444c-8e7e-5b7c16927ceb-certificates\") pod \"keda-operator-77f648d694-kd6w2\" (UID: \"50b5b7c8-b816-444c-8e7e-5b7c16927ceb\") " pod="openshift-keda/keda-operator-77f648d694-kd6w2"
Jul 20 10:03:21.482078 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.482044    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:21.541479 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.541403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f0c8d182-15f8-4eb1-90c5-0e775deffc23-certificates\") pod \"keda-admission-56f9798bb4-v9gzk\" (UID: \"f0c8d182-15f8-4eb1-90c5-0e775deffc23\") " pod="openshift-keda/keda-admission-56f9798bb4-v9gzk"
Jul 20 10:03:21.541616 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.541500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr79m\" (UniqueName: \"kubernetes.io/projected/f0c8d182-15f8-4eb1-90c5-0e775deffc23-kube-api-access-qr79m\") pod \"keda-admission-56f9798bb4-v9gzk\" (UID: \"f0c8d182-15f8-4eb1-90c5-0e775deffc23\") " pod="openshift-keda/keda-admission-56f9798bb4-v9gzk"
Jul 20 10:03:21.541616 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:03:21.541556    2580 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 20 10:03:21.541616 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:03:21.541585    2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-v9gzk: secret "keda-admission-webhooks-certs" not found
Jul 20 10:03:21.541757 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:03:21.541636    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0c8d182-15f8-4eb1-90c5-0e775deffc23-certificates podName:f0c8d182-15f8-4eb1-90c5-0e775deffc23 nodeName:}" failed. No retries permitted until 2026-07-20 10:03:22.041621294 +0000 UTC m=+414.090484923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f0c8d182-15f8-4eb1-90c5-0e775deffc23-certificates") pod "keda-admission-56f9798bb4-v9gzk" (UID: "f0c8d182-15f8-4eb1-90c5-0e775deffc23") : secret "keda-admission-webhooks-certs" not found
Jul 20 10:03:21.554078 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.554051    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr79m\" (UniqueName: \"kubernetes.io/projected/f0c8d182-15f8-4eb1-90c5-0e775deffc23-kube-api-access-qr79m\") pod \"keda-admission-56f9798bb4-v9gzk\" (UID: \"f0c8d182-15f8-4eb1-90c5-0e775deffc23\") " pod="openshift-keda/keda-admission-56f9798bb4-v9gzk"
Jul 20 10:03:21.582679 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.582653    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:21.668092 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.668059    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-kd6w2"
Jul 20 10:03:21.798700 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.798617    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-kd6w2"]
Jul 20 10:03:21.801480 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:03:21.801439    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b5b7c8_b816_444c_8e7e_5b7c16927ceb.slice/crio-399c4d543b41afc389bee97f9bbc7c2428e500ef31fe60c1ce4e11e852589120 WatchSource:0}: Error finding container 399c4d543b41afc389bee97f9bbc7c2428e500ef31fe60c1ce4e11e852589120: Status 404 returned error can't find the container with id 399c4d543b41afc389bee97f9bbc7c2428e500ef31fe60c1ce4e11e852589120
Jul 20 10:03:21.811762 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.811738    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:21.914484 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.914449    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:21.954225 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:21.954196    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-kd6w2" event={"ID":"50b5b7c8-b816-444c-8e7e-5b7c16927ceb","Type":"ContainerStarted","Data":"399c4d543b41afc389bee97f9bbc7c2428e500ef31fe60c1ce4e11e852589120"}
Jul 20 10:03:22.024181 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:22.024130    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:22.046520 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:22.046473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f0c8d182-15f8-4eb1-90c5-0e775deffc23-certificates\") pod \"keda-admission-56f9798bb4-v9gzk\" (UID: \"f0c8d182-15f8-4eb1-90c5-0e775deffc23\") " pod="openshift-keda/keda-admission-56f9798bb4-v9gzk"
Jul 20 10:03:22.051234 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:22.049995    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f0c8d182-15f8-4eb1-90c5-0e775deffc23-certificates\") pod \"keda-admission-56f9798bb4-v9gzk\" (UID: \"f0c8d182-15f8-4eb1-90c5-0e775deffc23\") " pod="openshift-keda/keda-admission-56f9798bb4-v9gzk"
Jul 20 10:03:22.169705 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:22.169651    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:22.268384 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:22.268348    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:22.308438 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:22.308358    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-v9gzk"
Jul 20 10:03:22.443573 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:22.443419    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-v9gzk"]
Jul 20 10:03:22.445986 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:03:22.445958    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c8d182_15f8_4eb1_90c5_0e775deffc23.slice/crio-933b52936a34bfc3a2f9bdf0bc0db139a7f49cc38aa9a0d2b232ceb2a19f4ea3 WatchSource:0}: Error finding container 933b52936a34bfc3a2f9bdf0bc0db139a7f49cc38aa9a0d2b232ceb2a19f4ea3: Status 404 returned error can't find the container with id 933b52936a34bfc3a2f9bdf0bc0db139a7f49cc38aa9a0d2b232ceb2a19f4ea3
Jul 20 10:03:22.959836 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:22.959793    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-v9gzk" event={"ID":"f0c8d182-15f8-4eb1-90c5-0e775deffc23","Type":"ContainerStarted","Data":"933b52936a34bfc3a2f9bdf0bc0db139a7f49cc38aa9a0d2b232ceb2a19f4ea3"}
Jul 20 10:03:23.966377 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:23.966326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-v9gzk" event={"ID":"f0c8d182-15f8-4eb1-90c5-0e775deffc23","Type":"ContainerStarted","Data":"c1716f90a2a6e8e658d65c2c4f74332518ddcc6877b3c68296dbc8fc1e943521"}
Jul 20 10:03:23.966952 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:23.966420    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-v9gzk"
Jul 20 10:03:23.987535 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:23.987487    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-v9gzk" podStartSLOduration=1.6279141780000002 podStartE2EDuration="2.987470456s" totalImagesPullingTime="1.359556278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:03:23.985601289 +0000 UTC m=+416.034464935" watchObservedRunningTime="2026-07-20 10:03:23.987470456 +0000 UTC m=+416.036334104"
Jul 20 10:03:25.975364 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:25.975329    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-kd6w2" event={"ID":"50b5b7c8-b816-444c-8e7e-5b7c16927ceb","Type":"ContainerStarted","Data":"95c07a70fe715caff665718446fe62014cf0e349165cfdc90e3c6aeb5901c3b8"}
Jul 20 10:03:25.975848 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:25.975503    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-kd6w2"
Jul 20 10:03:25.997900 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:25.997839    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-kd6w2" podStartSLOduration=2.663904089 podStartE2EDuration="5.997817537s" totalImagesPullingTime="3.333913448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:03:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:03:25.996571376 +0000 UTC m=+418.045435030" watchObservedRunningTime="2026-07-20 10:03:25.997817537 +0000 UTC m=+418.046681188"
Jul 20 10:03:41.957005 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:41.956974    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kxrbc"
Jul 20 10:03:44.973466 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:44.973438    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-v9gzk"
Jul 20 10:03:46.981838 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:03:46.981809    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-kd6w2"
Jul 20 10:04:13.210944 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:13.210910    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:13.310852 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:13.310822    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:15.253876 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:15.253839    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:15.326544 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:15.326499    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:36.266825 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:36.266787    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:36.426131 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:36.426075    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:36.616399 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:36.616309    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-bn4s5"]
Jul 20 10:04:36.619317 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:36.619295    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-bn4s5"
Jul 20 10:04:36.622955 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:36.622518    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 20 10:04:36.622955 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:36.622520    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 20 10:04:36.623130 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:36.623082    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-7wwmc\""
Jul 20 10:04:36.623217 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:36.623156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:36.623738 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:36.623367    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 20 10:04:36.637892 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:36.637870    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-bn4s5"]
Jul 20 10:04:36.727018 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:36.726979    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg2bn\" (UniqueName: \"kubernetes.io/projected/d774081a-91db-4bd2-9f8c-5616fc11d52a-kube-api-access-mg2bn\") pod \"seaweedfs-64568bcd49-bn4s5\" (UID: \"d774081a-91db-4bd2-9f8c-5616fc11d52a\") " pod="kserve/seaweedfs-64568bcd49-bn4s5"
Jul 20 10:04:36.727018 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:36.727023    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d774081a-91db-4bd2-9f8c-5616fc11d52a-data\") pod \"seaweedfs-64568bcd49-bn4s5\" (UID: \"d774081a-91db-4bd2-9f8c-5616fc11d52a\") " pod="kserve/seaweedfs-64568bcd49-bn4s5"
Jul 20 10:04:36.742471 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:36.742433    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:36.827764 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:36.827727    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg2bn\" (UniqueName: \"kubernetes.io/projected/d774081a-91db-4bd2-9f8c-5616fc11d52a-kube-api-access-mg2bn\") pod \"seaweedfs-64568bcd49-bn4s5\" (UID: \"d774081a-91db-4bd2-9f8c-5616fc11d52a\") " pod="kserve/seaweedfs-64568bcd49-bn4s5"
Jul 20 10:04:36.827914 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:36.827776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d774081a-91db-4bd2-9f8c-5616fc11d52a-data\") pod \"seaweedfs-64568bcd49-bn4s5\" (UID: \"d774081a-91db-4bd2-9f8c-5616fc11d52a\") " pod="kserve/seaweedfs-64568bcd49-bn4s5"
Jul 20 10:04:36.828142 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:36.828127    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d774081a-91db-4bd2-9f8c-5616fc11d52a-data\") pod \"seaweedfs-64568bcd49-bn4s5\" (UID: \"d774081a-91db-4bd2-9f8c-5616fc11d52a\") " pod="kserve/seaweedfs-64568bcd49-bn4s5"
Jul 20 10:04:36.848204 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:36.848156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg2bn\" (UniqueName: \"kubernetes.io/projected/d774081a-91db-4bd2-9f8c-5616fc11d52a-kube-api-access-mg2bn\") pod \"seaweedfs-64568bcd49-bn4s5\" (UID: \"d774081a-91db-4bd2-9f8c-5616fc11d52a\") " pod="kserve/seaweedfs-64568bcd49-bn4s5"
Jul 20 10:04:36.930901 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:36.930802    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-bn4s5"
Jul 20 10:04:37.066052 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:37.066021    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-bn4s5"]
Jul 20 10:04:37.069284 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:04:37.069243    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd774081a_91db_4bd2_9f8c_5616fc11d52a.slice/crio-2841c04e3fa9022c4ac9d0462b91256f8054bc78a788e73b88cd0147146de5ab WatchSource:0}: Error finding container 2841c04e3fa9022c4ac9d0462b91256f8054bc78a788e73b88cd0147146de5ab: Status 404 returned error can't find the container with id 2841c04e3fa9022c4ac9d0462b91256f8054bc78a788e73b88cd0147146de5ab
Jul 20 10:04:37.238058 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:37.238024    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-bn4s5" event={"ID":"d774081a-91db-4bd2-9f8c-5616fc11d52a","Type":"ContainerStarted","Data":"2841c04e3fa9022c4ac9d0462b91256f8054bc78a788e73b88cd0147146de5ab"}
Jul 20 10:04:37.299626 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:37.299593    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:37.402495 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:37.402465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:41.256145 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:41.256110    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-bn4s5" event={"ID":"d774081a-91db-4bd2-9f8c-5616fc11d52a","Type":"ContainerStarted","Data":"43a272dcaab0e2b33c6711bbb70a5254a305d627233e8e59a2860f433faae228"}
Jul 20 10:04:41.256694 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:41.256236    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-bn4s5"
Jul 20 10:04:41.281030 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:41.280972    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-bn4s5" podStartSLOduration=1.781749274 podStartE2EDuration="5.280954077s" totalImagesPullingTime="3.499204803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:04:41.279744064 +0000 UTC m=+493.328607712" watchObservedRunningTime="2026-07-20 10:04:41.280954077 +0000 UTC m=+493.329817724"
Jul 20 10:04:47.262550 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:47.262515    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-bn4s5"
Jul 20 10:04:59.687568 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:04:59.687531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:10.286142 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:10.286104    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:10.442775 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:10.442743    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:17.894234 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:17.894197    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:20.230383 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.230296    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:20.438050 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.438014    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:20.496142 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.496072    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb"]
Jul 20 10:05:20.499313 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.499298    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb"
Jul 20 10:05:20.501238 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.501214    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 20 10:05:20.501712 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.501694    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 20 10:05:20.501845 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.501714    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kznnl\""
Jul 20 10:05:20.510045 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.509956    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb"]
Jul 20 10:05:20.567699 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.567669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86ef8619-4ff9-4b94-ac0b-5a6a892eaa17-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb\" (UID: \"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb"
Jul 20 10:05:20.567853 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.567709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86ef8619-4ff9-4b94-ac0b-5a6a892eaa17-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb\" (UID: \"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb"
Jul 20 10:05:20.567853 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.567731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4tbb\" (UniqueName: \"kubernetes.io/projected/86ef8619-4ff9-4b94-ac0b-5a6a892eaa17-kube-api-access-x4tbb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb\" (UID: \"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb"
Jul 20 10:05:20.668877 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.668841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86ef8619-4ff9-4b94-ac0b-5a6a892eaa17-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb\" (UID: \"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb"
Jul 20 10:05:20.668877 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.668880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4tbb\" (UniqueName: \"kubernetes.io/projected/86ef8619-4ff9-4b94-ac0b-5a6a892eaa17-kube-api-access-x4tbb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb\" (UID: \"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb"
Jul 20 10:05:20.669083 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.668935    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86ef8619-4ff9-4b94-ac0b-5a6a892eaa17-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb\" (UID: \"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb"
Jul 20 10:05:20.669242 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.669223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86ef8619-4ff9-4b94-ac0b-5a6a892eaa17-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb\" (UID: \"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb"
Jul 20 10:05:20.669288 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.669254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86ef8619-4ff9-4b94-ac0b-5a6a892eaa17-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb\" (UID: \"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb"
Jul 20 10:05:20.678949 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.678916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4tbb\" (UniqueName: \"kubernetes.io/projected/86ef8619-4ff9-4b94-ac0b-5a6a892eaa17-kube-api-access-x4tbb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb\" (UID: \"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb"
Jul 20 10:05:20.814437 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.814343    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb"
Jul 20 10:05:20.944065 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:20.944040    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb"]
Jul 20 10:05:20.946679 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:05:20.946653    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ef8619_4ff9_4b94_ac0b_5a6a892eaa17.slice/crio-8a5bcf080dab3fd7b21b3630e6b059a4534c181235090c4d0cc268850836940e WatchSource:0}: Error finding container 8a5bcf080dab3fd7b21b3630e6b059a4534c181235090c4d0cc268850836940e: Status 404 returned error can't find the container with id 8a5bcf080dab3fd7b21b3630e6b059a4534c181235090c4d0cc268850836940e
Jul 20 10:05:21.407985 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:21.407953    2580 generic.go:356] "Generic (PLEG): container finished" podID="86ef8619-4ff9-4b94-ac0b-5a6a892eaa17" containerID="5a8dd29af66c0bae8c319529bda43055aacd9feb07a39445c8d5879f2c587844" exitCode=0
Jul 20 10:05:21.408594 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:21.408042    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb" event={"ID":"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17","Type":"ContainerDied","Data":"5a8dd29af66c0bae8c319529bda43055aacd9feb07a39445c8d5879f2c587844"}
Jul 20 10:05:21.408594 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:21.408091    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb" event={"ID":"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17","Type":"ContainerStarted","Data":"8a5bcf080dab3fd7b21b3630e6b059a4534c181235090c4d0cc268850836940e"}
Jul 20 10:05:24.423126 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:24.423093    2580 generic.go:356] "Generic (PLEG): container finished" podID="86ef8619-4ff9-4b94-ac0b-5a6a892eaa17" containerID="e3083b26ec068ecd40e6a8c230e65c7fb7380f7e53548f31333e1251df24f249" exitCode=0
Jul 20 10:05:24.423564 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:24.423215    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb" event={"ID":"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17","Type":"ContainerDied","Data":"e3083b26ec068ecd40e6a8c230e65c7fb7380f7e53548f31333e1251df24f249"}
Jul 20 10:05:25.429617 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:25.429587    2580 generic.go:356] "Generic (PLEG): container finished" podID="86ef8619-4ff9-4b94-ac0b-5a6a892eaa17" containerID="24509b31177eff43f0b1aa84c5203a64ec0ac6b553c9e5ac533a4906657d69c8" exitCode=0
Jul 20 10:05:25.429999 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:25.429653    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb" event={"ID":"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17","Type":"ContainerDied","Data":"24509b31177eff43f0b1aa84c5203a64ec0ac6b553c9e5ac533a4906657d69c8"}
Jul 20 10:05:26.562778 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:26.562756    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb"
Jul 20 10:05:26.608388 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:26.608353    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4tbb\" (UniqueName: \"kubernetes.io/projected/86ef8619-4ff9-4b94-ac0b-5a6a892eaa17-kube-api-access-x4tbb\") pod \"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17\" (UID: \"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17\") "
Jul 20 10:05:26.608388 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:26.608387    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86ef8619-4ff9-4b94-ac0b-5a6a892eaa17-bundle\") pod \"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17\" (UID: \"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17\") "
Jul 20 10:05:26.608626 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:26.608441    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86ef8619-4ff9-4b94-ac0b-5a6a892eaa17-util\") pod \"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17\" (UID: \"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17\") "
Jul 20 10:05:26.609144 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:26.609117    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ef8619-4ff9-4b94-ac0b-5a6a892eaa17-bundle" (OuterVolumeSpecName: "bundle") pod "86ef8619-4ff9-4b94-ac0b-5a6a892eaa17" (UID: "86ef8619-4ff9-4b94-ac0b-5a6a892eaa17"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:05:26.610500 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:26.610478    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ef8619-4ff9-4b94-ac0b-5a6a892eaa17-kube-api-access-x4tbb" (OuterVolumeSpecName: "kube-api-access-x4tbb") pod "86ef8619-4ff9-4b94-ac0b-5a6a892eaa17" (UID: "86ef8619-4ff9-4b94-ac0b-5a6a892eaa17"). InnerVolumeSpecName "kube-api-access-x4tbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:05:26.613613 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:26.613594    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ef8619-4ff9-4b94-ac0b-5a6a892eaa17-util" (OuterVolumeSpecName: "util") pod "86ef8619-4ff9-4b94-ac0b-5a6a892eaa17" (UID: "86ef8619-4ff9-4b94-ac0b-5a6a892eaa17"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:05:26.709667 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:26.709637    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86ef8619-4ff9-4b94-ac0b-5a6a892eaa17-util\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:05:26.709805 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:26.709692    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4tbb\" (UniqueName: \"kubernetes.io/projected/86ef8619-4ff9-4b94-ac0b-5a6a892eaa17-kube-api-access-x4tbb\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:05:26.709805 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:26.709706    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86ef8619-4ff9-4b94-ac0b-5a6a892eaa17-bundle\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:05:27.440054 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:27.440014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb" event={"ID":"86ef8619-4ff9-4b94-ac0b-5a6a892eaa17","Type":"ContainerDied","Data":"8a5bcf080dab3fd7b21b3630e6b059a4534c181235090c4d0cc268850836940e"}
Jul 20 10:05:27.440250 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:27.440066    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q7cxb"
Jul 20 10:05:27.440250 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:27.440073    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a5bcf080dab3fd7b21b3630e6b059a4534c181235090c4d0cc268850836940e"
Jul 20 10:05:31.514647 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:31.514609    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:32.295701 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:32.295656    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:34.159868 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.159840    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qj7fw"]
Jul 20 10:05:34.160257 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.160157    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86ef8619-4ff9-4b94-ac0b-5a6a892eaa17" containerName="extract"
Jul 20 10:05:34.160257 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.160190    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="86ef8619-4ff9-4b94-ac0b-5a6a892eaa17" containerName="extract"
Jul 20 10:05:34.160257 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.160208    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86ef8619-4ff9-4b94-ac0b-5a6a892eaa17" containerName="pull"
Jul 20 10:05:34.160257 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.160214    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="86ef8619-4ff9-4b94-ac0b-5a6a892eaa17" containerName="pull"
Jul 20 10:05:34.160257 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.160223    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86ef8619-4ff9-4b94-ac0b-5a6a892eaa17" containerName="util"
Jul 20 10:05:34.160257 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.160228    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="86ef8619-4ff9-4b94-ac0b-5a6a892eaa17" containerName="util"
Jul 20 10:05:34.160438 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.160298    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="86ef8619-4ff9-4b94-ac0b-5a6a892eaa17" containerName="extract"
Jul 20 10:05:34.163953 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.163937    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qj7fw"
Jul 20 10:05:34.171489 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.171453    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 20 10:05:34.171591 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.171530    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vmrqf\""
Jul 20 10:05:34.171675 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.171656    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 20 10:05:34.216142 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.216113    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qj7fw"]
Jul 20 10:05:34.267764 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.267720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r45jq\" (UniqueName: \"kubernetes.io/projected/8f3cf760-443c-4a11-9506-40a2f7f915bd-kube-api-access-r45jq\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qj7fw\" (UID: \"8f3cf760-443c-4a11-9506-40a2f7f915bd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qj7fw"
Jul 20 10:05:34.267950 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.267807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f3cf760-443c-4a11-9506-40a2f7f915bd-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qj7fw\" (UID: \"8f3cf760-443c-4a11-9506-40a2f7f915bd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qj7fw"
Jul 20 10:05:34.369067 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.369012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f3cf760-443c-4a11-9506-40a2f7f915bd-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qj7fw\" (UID: \"8f3cf760-443c-4a11-9506-40a2f7f915bd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qj7fw"
Jul 20 10:05:34.369286 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.369118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r45jq\" (UniqueName: \"kubernetes.io/projected/8f3cf760-443c-4a11-9506-40a2f7f915bd-kube-api-access-r45jq\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qj7fw\" (UID: \"8f3cf760-443c-4a11-9506-40a2f7f915bd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qj7fw"
Jul 20 10:05:34.369514 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.369490    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f3cf760-443c-4a11-9506-40a2f7f915bd-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qj7fw\" (UID: \"8f3cf760-443c-4a11-9506-40a2f7f915bd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qj7fw"
Jul 20 10:05:34.380492 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.380473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r45jq\" (UniqueName: \"kubernetes.io/projected/8f3cf760-443c-4a11-9506-40a2f7f915bd-kube-api-access-r45jq\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qj7fw\" (UID: \"8f3cf760-443c-4a11-9506-40a2f7f915bd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qj7fw"
Jul 20 10:05:34.472864 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.472837    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qj7fw"
Jul 20 10:05:34.620712 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:34.620688    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qj7fw"]
Jul 20 10:05:34.623417 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:05:34.623386    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3cf760_443c_4a11_9506_40a2f7f915bd.slice/crio-21cc1e62c4a7663ab195e9683f55e5b123dfc3f85864a974c93f5e3854d3632e WatchSource:0}: Error finding container 21cc1e62c4a7663ab195e9683f55e5b123dfc3f85864a974c93f5e3854d3632e: Status 404 returned error can't find the container with id 21cc1e62c4a7663ab195e9683f55e5b123dfc3f85864a974c93f5e3854d3632e
Jul 20 10:05:35.473997 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:35.473957    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qj7fw" event={"ID":"8f3cf760-443c-4a11-9506-40a2f7f915bd","Type":"ContainerStarted","Data":"21cc1e62c4a7663ab195e9683f55e5b123dfc3f85864a974c93f5e3854d3632e"}
Jul 20 10:05:35.958031 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:35.957994    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:36.176100 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:36.176068    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:36.328916 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:36.328872    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:36.482875 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:36.482786    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qj7fw" event={"ID":"8f3cf760-443c-4a11-9506-40a2f7f915bd","Type":"ContainerStarted","Data":"14a7868edf418ab2759a03fadade5b36163e4a32b0d31a06f582e2b77bd7c53e"}
Jul 20 10:05:37.101203 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:37.101157    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:37.402663 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:37.402569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:39.693739 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:39.693704    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:40.047485 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:40.047427    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qj7fw" podStartSLOduration=4.460760092 podStartE2EDuration="6.047408504s" totalImagesPullingTime="1.586648412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:05:36.552057414 +0000 UTC m=+548.600921066" watchObservedRunningTime="2026-07-20 10:05:40.047408504 +0000 UTC m=+552.096272163"
Jul 20 10:05:40.048540 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:40.048508    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6jxq9"]
Jul 20 10:05:40.051825 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:40.051805    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6jxq9"
Jul 20 10:05:40.062717 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:40.062700    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-t997k\""
Jul 20 10:05:40.081758 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:40.081733    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 20 10:05:40.081758 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:40.081750    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 20 10:05:40.091758 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:40.091738    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6jxq9"]
Jul 20 10:05:40.112119 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:40.112087    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3320a2e-0d2b-4a5a-970a-6de37d27b9b6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6jxq9\" (UID: \"b3320a2e-0d2b-4a5a-970a-6de37d27b9b6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6jxq9"
Jul 20 10:05:40.112259 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:40.112201    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74dfb\" (UniqueName: \"kubernetes.io/projected/b3320a2e-0d2b-4a5a-970a-6de37d27b9b6-kube-api-access-74dfb\") pod \"cert-manager-cainjector-559cdd8c87-6jxq9\" (UID: \"b3320a2e-0d2b-4a5a-970a-6de37d27b9b6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6jxq9"
Jul 20 10:05:40.213323 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:40.213290    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74dfb\" (UniqueName: \"kubernetes.io/projected/b3320a2e-0d2b-4a5a-970a-6de37d27b9b6-kube-api-access-74dfb\") pod \"cert-manager-cainjector-559cdd8c87-6jxq9\" (UID: \"b3320a2e-0d2b-4a5a-970a-6de37d27b9b6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6jxq9"
Jul 20 10:05:40.213503 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:40.213352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3320a2e-0d2b-4a5a-970a-6de37d27b9b6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6jxq9\" (UID: \"b3320a2e-0d2b-4a5a-970a-6de37d27b9b6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6jxq9"
Jul 20 10:05:40.246422 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:40.246393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74dfb\" (UniqueName: \"kubernetes.io/projected/b3320a2e-0d2b-4a5a-970a-6de37d27b9b6-kube-api-access-74dfb\") pod \"cert-manager-cainjector-559cdd8c87-6jxq9\" (UID: \"b3320a2e-0d2b-4a5a-970a-6de37d27b9b6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6jxq9"
Jul 20 10:05:40.248289 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:40.248273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3320a2e-0d2b-4a5a-970a-6de37d27b9b6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6jxq9\" (UID: \"b3320a2e-0d2b-4a5a-970a-6de37d27b9b6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6jxq9"
Jul 20 10:05:40.317982 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:40.317872    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:40.360684 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:40.360656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6jxq9"
Jul 20 10:05:40.521006 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:40.520975    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6jxq9"]
Jul 20 10:05:40.528746 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:05:40.528717    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3320a2e_0d2b_4a5a_970a_6de37d27b9b6.slice/crio-cbe3c1c09bf719f60c0361ae3694322e88b709ce8d2edfcffb1d8ab930065444 WatchSource:0}: Error finding container cbe3c1c09bf719f60c0361ae3694322e88b709ce8d2edfcffb1d8ab930065444: Status 404 returned error can't find the container with id cbe3c1c09bf719f60c0361ae3694322e88b709ce8d2edfcffb1d8ab930065444
Jul 20 10:05:41.505534 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:41.505493    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6jxq9" event={"ID":"b3320a2e-0d2b-4a5a-970a-6de37d27b9b6","Type":"ContainerStarted","Data":"cbe3c1c09bf719f60c0361ae3694322e88b709ce8d2edfcffb1d8ab930065444"}
Jul 20 10:05:41.989906 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:41.989871    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:42.438742 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:42.438642    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:42.617956 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:42.617892    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:42.726651 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:42.726625    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck"]
Jul 20 10:05:42.730185 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:42.730148    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck"
Jul 20 10:05:42.734147 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:42.734125    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kznnl\""
Jul 20 10:05:42.734535 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:42.734512    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 20 10:05:42.734635 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:42.734539    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 20 10:05:42.763831 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:42.763804    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck"]
Jul 20 10:05:42.841842 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:42.838864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/526fd995-9e6e-485b-9e2e-44f8124f3449-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck\" (UID: \"526fd995-9e6e-485b-9e2e-44f8124f3449\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck"
Jul 20 10:05:42.841842 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:42.839083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/526fd995-9e6e-485b-9e2e-44f8124f3449-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck\" (UID: \"526fd995-9e6e-485b-9e2e-44f8124f3449\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck"
Jul 20 10:05:42.841842 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:42.839144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kccb6\" (UniqueName: \"kubernetes.io/projected/526fd995-9e6e-485b-9e2e-44f8124f3449-kube-api-access-kccb6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck\" (UID: \"526fd995-9e6e-485b-9e2e-44f8124f3449\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck"
Jul 20 10:05:42.940616 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:42.940575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/526fd995-9e6e-485b-9e2e-44f8124f3449-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck\" (UID: \"526fd995-9e6e-485b-9e2e-44f8124f3449\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck"
Jul 20 10:05:42.940807 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:42.940646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kccb6\" (UniqueName: \"kubernetes.io/projected/526fd995-9e6e-485b-9e2e-44f8124f3449-kube-api-access-kccb6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck\" (UID: \"526fd995-9e6e-485b-9e2e-44f8124f3449\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck"
Jul 20 10:05:42.940807 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:42.940697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/526fd995-9e6e-485b-9e2e-44f8124f3449-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck\" (UID: \"526fd995-9e6e-485b-9e2e-44f8124f3449\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck"
Jul 20 10:05:42.941061 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:42.941038    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/526fd995-9e6e-485b-9e2e-44f8124f3449-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck\" (UID: \"526fd995-9e6e-485b-9e2e-44f8124f3449\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck"
Jul 20 10:05:42.941142 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:42.941059    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/526fd995-9e6e-485b-9e2e-44f8124f3449-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck\" (UID: \"526fd995-9e6e-485b-9e2e-44f8124f3449\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck"
Jul 20 10:05:42.961035 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:42.961005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kccb6\" (UniqueName: \"kubernetes.io/projected/526fd995-9e6e-485b-9e2e-44f8124f3449-kube-api-access-kccb6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck\" (UID: \"526fd995-9e6e-485b-9e2e-44f8124f3449\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck"
Jul 20 10:05:43.041640 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:43.041556    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck"
Jul 20 10:05:43.218864 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:43.218834    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck"]
Jul 20 10:05:43.221743 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:05:43.221712    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526fd995_9e6e_485b_9e2e_44f8124f3449.slice/crio-6bd86436475c5ea9011d7a9ee30714432e48a505bfba1796752aaf69da14ff6e WatchSource:0}: Error finding container 6bd86436475c5ea9011d7a9ee30714432e48a505bfba1796752aaf69da14ff6e: Status 404 returned error can't find the container with id 6bd86436475c5ea9011d7a9ee30714432e48a505bfba1796752aaf69da14ff6e
Jul 20 10:05:43.519424 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:43.519389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6jxq9" event={"ID":"b3320a2e-0d2b-4a5a-970a-6de37d27b9b6","Type":"ContainerStarted","Data":"7f4761083b4917dab2ccf21312be4acc5ea218b9d7954d855f73fecb46348d86"}
Jul 20 10:05:43.520836 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:43.520812    2580 generic.go:356] "Generic (PLEG): container finished" podID="526fd995-9e6e-485b-9e2e-44f8124f3449" containerID="653a5500620b80cda1536e6d7893b45d22a81a8b73f2ac21d26f92ec509bf965" exitCode=0
Jul 20 10:05:43.520955 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:43.520851    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck" event={"ID":"526fd995-9e6e-485b-9e2e-44f8124f3449","Type":"ContainerDied","Data":"653a5500620b80cda1536e6d7893b45d22a81a8b73f2ac21d26f92ec509bf965"}
Jul 20 10:05:43.520955 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:43.520882    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck" event={"ID":"526fd995-9e6e-485b-9e2e-44f8124f3449","Type":"ContainerStarted","Data":"6bd86436475c5ea9011d7a9ee30714432e48a505bfba1796752aaf69da14ff6e"}
Jul 20 10:05:43.597837 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:43.597727    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6jxq9" podStartSLOduration=2.011087903 podStartE2EDuration="4.597708097s" totalImagesPullingTime="2.586620194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:05:43.593087179 +0000 UTC m=+555.641950828" watchObservedRunningTime="2026-07-20 10:05:43.597708097 +0000 UTC m=+555.646571744"
Jul 20 10:05:45.517209 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:45.517131    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:45.868597 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:45.868563    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:46.536566 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:46.536538    2580 generic.go:356] "Generic (PLEG): container finished" podID="526fd995-9e6e-485b-9e2e-44f8124f3449" containerID="c7c62be80613df459807f8400ebdc3a35a3a6d99d1e6a414feef6feab7064dd8" exitCode=0
Jul 20 10:05:46.536950 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:46.536627    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck" event={"ID":"526fd995-9e6e-485b-9e2e-44f8124f3449","Type":"ContainerDied","Data":"c7c62be80613df459807f8400ebdc3a35a3a6d99d1e6a414feef6feab7064dd8"}
Jul 20 10:05:46.793833 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:46.793744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:47.542912 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:47.542876    2580 generic.go:356] "Generic (PLEG): container finished" podID="526fd995-9e6e-485b-9e2e-44f8124f3449" containerID="1b333480cac47cb230226b5a551c492f3db7d64a65dab2683980f2d870afcfd7" exitCode=0
Jul 20 10:05:47.543279 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:47.542957    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck" event={"ID":"526fd995-9e6e-485b-9e2e-44f8124f3449","Type":"ContainerDied","Data":"1b333480cac47cb230226b5a551c492f3db7d64a65dab2683980f2d870afcfd7"}
Jul 20 10:05:48.673375 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:48.673350    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck"
Jul 20 10:05:48.783922 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:48.783898    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kccb6\" (UniqueName: \"kubernetes.io/projected/526fd995-9e6e-485b-9e2e-44f8124f3449-kube-api-access-kccb6\") pod \"526fd995-9e6e-485b-9e2e-44f8124f3449\" (UID: \"526fd995-9e6e-485b-9e2e-44f8124f3449\") "
Jul 20 10:05:48.784129 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:48.783956    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/526fd995-9e6e-485b-9e2e-44f8124f3449-bundle\") pod \"526fd995-9e6e-485b-9e2e-44f8124f3449\" (UID: \"526fd995-9e6e-485b-9e2e-44f8124f3449\") "
Jul 20 10:05:48.784129 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:48.783983    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/526fd995-9e6e-485b-9e2e-44f8124f3449-util\") pod \"526fd995-9e6e-485b-9e2e-44f8124f3449\" (UID: \"526fd995-9e6e-485b-9e2e-44f8124f3449\") "
Jul 20 10:05:48.784368 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:48.784342    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/526fd995-9e6e-485b-9e2e-44f8124f3449-bundle" (OuterVolumeSpecName: "bundle") pod "526fd995-9e6e-485b-9e2e-44f8124f3449" (UID: "526fd995-9e6e-485b-9e2e-44f8124f3449"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:05:48.786186 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:48.786139    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/526fd995-9e6e-485b-9e2e-44f8124f3449-kube-api-access-kccb6" (OuterVolumeSpecName: "kube-api-access-kccb6") pod "526fd995-9e6e-485b-9e2e-44f8124f3449" (UID: "526fd995-9e6e-485b-9e2e-44f8124f3449"). InnerVolumeSpecName "kube-api-access-kccb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:05:48.840504 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:48.840420    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/526fd995-9e6e-485b-9e2e-44f8124f3449-util" (OuterVolumeSpecName: "util") pod "526fd995-9e6e-485b-9e2e-44f8124f3449" (UID: "526fd995-9e6e-485b-9e2e-44f8124f3449"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:05:48.884557 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:48.884514    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kccb6\" (UniqueName: \"kubernetes.io/projected/526fd995-9e6e-485b-9e2e-44f8124f3449-kube-api-access-kccb6\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:05:48.884557 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:48.884551    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/526fd995-9e6e-485b-9e2e-44f8124f3449-bundle\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:05:48.884557 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:48.884562    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/526fd995-9e6e-485b-9e2e-44f8124f3449-util\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:05:49.554298 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:49.554266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck" event={"ID":"526fd995-9e6e-485b-9e2e-44f8124f3449","Type":"ContainerDied","Data":"6bd86436475c5ea9011d7a9ee30714432e48a505bfba1796752aaf69da14ff6e"}
Jul 20 10:05:49.554472 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:49.554308    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4wbck"
Jul 20 10:05:49.554472 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:49.554312    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bd86436475c5ea9011d7a9ee30714432e48a505bfba1796752aaf69da14ff6e"
Jul 20 10:05:53.564374 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:53.564335    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:53.801331 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:05:53.801295    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:05.654196 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:05.654151    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:11.236310 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.236260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:11.245514 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.245491    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"]
Jul 20 10:06:11.245777 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.245766    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="526fd995-9e6e-485b-9e2e-44f8124f3449" containerName="pull"
Jul 20 10:06:11.245821 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.245778    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="526fd995-9e6e-485b-9e2e-44f8124f3449" containerName="pull"
Jul 20 10:06:11.245821 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.245795    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="526fd995-9e6e-485b-9e2e-44f8124f3449" containerName="util"
Jul 20 10:06:11.245821 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.245801    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="526fd995-9e6e-485b-9e2e-44f8124f3449" containerName="util"
Jul 20 10:06:11.245821 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.245810    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="526fd995-9e6e-485b-9e2e-44f8124f3449" containerName="extract"
Jul 20 10:06:11.245821 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.245816    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="526fd995-9e6e-485b-9e2e-44f8124f3449" containerName="extract"
Jul 20 10:06:11.245967 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.245865    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="526fd995-9e6e-485b-9e2e-44f8124f3449" containerName="extract"
Jul 20 10:06:11.251677 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.251660    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.253689 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.253659    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 20 10:06:11.253821 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.253692    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 20 10:06:11.253821 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.253658    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 20 10:06:11.253936 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.253923    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 20 10:06:11.253989 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.253954    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jrzkr\""
Jul 20 10:06:11.254041 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.254004    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 20 10:06:11.254041 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.254014    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 20 10:06:11.260841 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.260818    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"]
Jul 20 10:06:11.332572 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.332543    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:11.369178 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.369132    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e9300ef-83e4-4b50-947f-06b61d0ec232-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.369322 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.369196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e9300ef-83e4-4b50-947f-06b61d0ec232-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.369322 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.369280    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgfpt\" (UniqueName: \"kubernetes.io/projected/6e9300ef-83e4-4b50-947f-06b61d0ec232-kube-api-access-qgfpt\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.369322 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.369308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e9300ef-83e4-4b50-947f-06b61d0ec232-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.369434 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.369326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e9300ef-83e4-4b50-947f-06b61d0ec232-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.369434 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.369342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e9300ef-83e4-4b50-947f-06b61d0ec232-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.369434 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.369378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e9300ef-83e4-4b50-947f-06b61d0ec232-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.470688 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.470651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e9300ef-83e4-4b50-947f-06b61d0ec232-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.470896 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.470693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e9300ef-83e4-4b50-947f-06b61d0ec232-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.470896 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.470739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e9300ef-83e4-4b50-947f-06b61d0ec232-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.470896 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.470770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e9300ef-83e4-4b50-947f-06b61d0ec232-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.470896 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.470819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgfpt\" (UniqueName: \"kubernetes.io/projected/6e9300ef-83e4-4b50-947f-06b61d0ec232-kube-api-access-qgfpt\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.470896 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.470840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e9300ef-83e4-4b50-947f-06b61d0ec232-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.470896 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.470866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e9300ef-83e4-4b50-947f-06b61d0ec232-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.471813 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.471784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e9300ef-83e4-4b50-947f-06b61d0ec232-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.473330 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.473307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e9300ef-83e4-4b50-947f-06b61d0ec232-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.473438 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.473366    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e9300ef-83e4-4b50-947f-06b61d0ec232-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.473547 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.473533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e9300ef-83e4-4b50-947f-06b61d0ec232-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.473744 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.473724    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e9300ef-83e4-4b50-947f-06b61d0ec232-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.479342 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.479319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e9300ef-83e4-4b50-947f-06b61d0ec232-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.479600 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.479585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgfpt\" (UniqueName: \"kubernetes.io/projected/6e9300ef-83e4-4b50-947f-06b61d0ec232-kube-api-access-qgfpt\") pod \"istiod-openshift-gateway-94bb8fbfd-zltzb\" (UID: \"6e9300ef-83e4-4b50-947f-06b61d0ec232\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.561730 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.561638    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:11.703592 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:11.703565    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"]
Jul 20 10:06:11.706974 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:06:11.706950    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9300ef_83e4_4b50_947f_06b61d0ec232.slice/crio-f66c7ed787f6b6d500e8a64207937a9dd3a9f7b3ade9e89db30ebf95622aff9d WatchSource:0}: Error finding container f66c7ed787f6b6d500e8a64207937a9dd3a9f7b3ade9e89db30ebf95622aff9d: Status 404 returned error can't find the container with id f66c7ed787f6b6d500e8a64207937a9dd3a9f7b3ade9e89db30ebf95622aff9d
Jul 20 10:06:12.554764 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:12.554723    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:12.657242 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:12.657198    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb" event={"ID":"6e9300ef-83e4-4b50-947f-06b61d0ec232","Type":"ContainerStarted","Data":"f66c7ed787f6b6d500e8a64207937a9dd3a9f7b3ade9e89db30ebf95622aff9d"}
Jul 20 10:06:14.023060 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:14.023018    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 20 10:06:14.023376 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:14.023090    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 20 10:06:14.656645 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:14.656598    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:14.668705 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:14.668679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb" event={"ID":"6e9300ef-83e4-4b50-947f-06b61d0ec232","Type":"ContainerStarted","Data":"c2d1166dee1151f4b48e4794e9e415a060f2b7dc76a8fc2fdd046fe0fac6dc47"}
Jul 20 10:06:14.668841 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:14.668787    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:14.692346 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:14.691639    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb" podStartSLOduration=1.378242528 podStartE2EDuration="3.691619095s" totalImagesPullingTime="2.313376567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:06:14.68959336 +0000 UTC m=+586.738457009" watchObservedRunningTime="2026-07-20 10:06:14.691619095 +0000 UTC m=+586.740482744"
Jul 20 10:06:14.758329 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:14.758294    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:15.674800 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:15.674760    2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-zltzb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 20 10:06:15.675251 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:15.674842    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb" podUID="6e9300ef-83e4-4b50-947f-06b61d0ec232" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:06:18.674216 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:18.674182    2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-zltzb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 20 10:06:18.674728 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:18.674241    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb" podUID="6e9300ef-83e4-4b50-947f-06b61d0ec232" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:06:21.674461 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:21.674424    2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-zltzb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 20 10:06:21.674901 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:21.674500    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb" podUID="6e9300ef-83e4-4b50-947f-06b61d0ec232" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:06:24.673860 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:24.673827    2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-zltzb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 20 10:06:24.674266 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:24.673883    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb" podUID="6e9300ef-83e4-4b50-947f-06b61d0ec232" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:06:27.674707 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:27.674666    2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-zltzb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 20 10:06:27.675097 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:27.674732    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb" podUID="6e9300ef-83e4-4b50-947f-06b61d0ec232" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:06:27.884125 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:27.884083    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:27.969944 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:27.969909    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:30.674470 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:30.674436    2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-zltzb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 20 10:06:30.674929 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:30.674497    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb" podUID="6e9300ef-83e4-4b50-947f-06b61d0ec232" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:06:33.674553 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:33.674515    2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-zltzb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 20 10:06:33.674929 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:33.674583    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb" podUID="6e9300ef-83e4-4b50-947f-06b61d0ec232" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:06:36.673849 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:36.673817    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zltzb"
Jul 20 10:06:37.395635 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.395603    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"]
Jul 20 10:06:37.400723 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.400707    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.418262 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.418235    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-4wnz2\""
Jul 20 10:06:37.429162 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.429123    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"]
Jul 20 10:06:37.489635 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.489595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06712755-7e3d-4b4c-9c07-3260be21366b-istio-data\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.489813 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.489644    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06712755-7e3d-4b4c-9c07-3260be21366b-credential-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.489813 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.489738    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06712755-7e3d-4b4c-9c07-3260be21366b-workload-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.489813 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.489779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js6nn\" (UniqueName: \"kubernetes.io/projected/06712755-7e3d-4b4c-9c07-3260be21366b-kube-api-access-js6nn\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.489937 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.489815    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06712755-7e3d-4b4c-9c07-3260be21366b-workload-certs\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.489937 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.489870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06712755-7e3d-4b4c-9c07-3260be21366b-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.489937 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.489898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06712755-7e3d-4b4c-9c07-3260be21366b-istio-envoy\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.489937 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.489917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06712755-7e3d-4b4c-9c07-3260be21366b-istio-token\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.489937 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.489932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06712755-7e3d-4b4c-9c07-3260be21366b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.507832 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.507784    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:37.590672 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.590638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06712755-7e3d-4b4c-9c07-3260be21366b-workload-certs\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.590672 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.590678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06712755-7e3d-4b4c-9c07-3260be21366b-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.590912 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.590700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06712755-7e3d-4b4c-9c07-3260be21366b-istio-envoy\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.590912 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.590722    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06712755-7e3d-4b4c-9c07-3260be21366b-istio-token\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.590912 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.590746    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06712755-7e3d-4b4c-9c07-3260be21366b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.590912 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.590775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06712755-7e3d-4b4c-9c07-3260be21366b-istio-data\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.590912 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.590809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06712755-7e3d-4b4c-9c07-3260be21366b-credential-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.590912 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.590879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06712755-7e3d-4b4c-9c07-3260be21366b-workload-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.591317 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.590915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js6nn\" (UniqueName: \"kubernetes.io/projected/06712755-7e3d-4b4c-9c07-3260be21366b-kube-api-access-js6nn\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.591317 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.591047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06712755-7e3d-4b4c-9c07-3260be21366b-workload-certs\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.591317 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.591125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06712755-7e3d-4b4c-9c07-3260be21366b-istio-data\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.591638 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.591326    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06712755-7e3d-4b4c-9c07-3260be21366b-credential-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.591638 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.591345    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06712755-7e3d-4b4c-9c07-3260be21366b-workload-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.591749 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.591712    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06712755-7e3d-4b4c-9c07-3260be21366b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.593081 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.593057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06712755-7e3d-4b4c-9c07-3260be21366b-istio-envoy\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.593245 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.593224    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06712755-7e3d-4b4c-9c07-3260be21366b-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.598083 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.598056    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:37.601698 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.601680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06712755-7e3d-4b4c-9c07-3260be21366b-istio-token\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.602067 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.602051    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js6nn\" (UniqueName: \"kubernetes.io/projected/06712755-7e3d-4b4c-9c07-3260be21366b-kube-api-access-js6nn\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-jfpwp\" (UID: \"06712755-7e3d-4b4c-9c07-3260be21366b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.710878 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.710844    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:37.858714 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:37.858685    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"]
Jul 20 10:06:37.862046 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:06:37.862010    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06712755_7e3d_4b4c_9c07_3260be21366b.slice/crio-c034e16a03887fa6bc043907c53fa457cd85185c3c785e884ef501b632ee5bce WatchSource:0}: Error finding container c034e16a03887fa6bc043907c53fa457cd85185c3c785e884ef501b632ee5bce: Status 404 returned error can't find the container with id c034e16a03887fa6bc043907c53fa457cd85185c3c785e884ef501b632ee5bce
Jul 20 10:06:38.775561 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:38.775526    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp" event={"ID":"06712755-7e3d-4b4c-9c07-3260be21366b","Type":"ContainerStarted","Data":"c034e16a03887fa6bc043907c53fa457cd85185c3c785e884ef501b632ee5bce"}
Jul 20 10:06:40.503745 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:40.503711    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 20 10:06:40.503978 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:40.503788    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 20 10:06:40.503978 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:40.503818    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 20 10:06:40.786982 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:40.786891    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp" event={"ID":"06712755-7e3d-4b4c-9c07-3260be21366b","Type":"ContainerStarted","Data":"f33529118c115cf2168c6ab7707cd0876fece2e16039240d3a010c8afd844a49"}
Jul 20 10:06:40.811715 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:40.811655    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp" podStartSLOduration=1.172305996 podStartE2EDuration="3.811633744s" totalImagesPullingTime="2.639327748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:06:40.808343341 +0000 UTC m=+612.857206988" watchObservedRunningTime="2026-07-20 10:06:40.811633744 +0000 UTC m=+612.860497392"
Jul 20 10:06:41.711380 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:41.711354    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:41.715999 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:41.715973    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:41.757252 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:41.757182    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:41.792460 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:41.792429    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:41.793508 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:41.793487    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:06:44.934439 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:44.934406    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:45.043540 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:45.043506    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:48.419068 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.419034    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst"]
Jul 20 10:06:48.426130 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.426111    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst"
Jul 20 10:06:48.430272 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.430241    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 20 10:06:48.430689 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.430668    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 20 10:06:48.430971 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.430954    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kznnl\""
Jul 20 10:06:48.457762 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.457733    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst"]
Jul 20 10:06:48.489753 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.489725    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e0ba9ce-b097-48e8-ad30-268f95740f34-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst\" (UID: \"0e0ba9ce-b097-48e8-ad30-268f95740f34\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst"
Jul 20 10:06:48.489933 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.489762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdvlr\" (UniqueName: \"kubernetes.io/projected/0e0ba9ce-b097-48e8-ad30-268f95740f34-kube-api-access-bdvlr\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst\" (UID: \"0e0ba9ce-b097-48e8-ad30-268f95740f34\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst"
Jul 20 10:06:48.489933 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.489795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e0ba9ce-b097-48e8-ad30-268f95740f34-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst\" (UID: \"0e0ba9ce-b097-48e8-ad30-268f95740f34\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst"
Jul 20 10:06:48.590737 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.590703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e0ba9ce-b097-48e8-ad30-268f95740f34-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst\" (UID: \"0e0ba9ce-b097-48e8-ad30-268f95740f34\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst"
Jul 20 10:06:48.590737 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.590743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdvlr\" (UniqueName: \"kubernetes.io/projected/0e0ba9ce-b097-48e8-ad30-268f95740f34-kube-api-access-bdvlr\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst\" (UID: \"0e0ba9ce-b097-48e8-ad30-268f95740f34\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst"
Jul 20 10:06:48.590966 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.590771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e0ba9ce-b097-48e8-ad30-268f95740f34-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst\" (UID: \"0e0ba9ce-b097-48e8-ad30-268f95740f34\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst"
Jul 20 10:06:48.591108 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.591093    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e0ba9ce-b097-48e8-ad30-268f95740f34-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst\" (UID: \"0e0ba9ce-b097-48e8-ad30-268f95740f34\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst"
Jul 20 10:06:48.591148 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.591118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e0ba9ce-b097-48e8-ad30-268f95740f34-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst\" (UID: \"0e0ba9ce-b097-48e8-ad30-268f95740f34\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst"
Jul 20 10:06:48.601405 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.601375    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87"]
Jul 20 10:06:48.604864 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.604848    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87"
Jul 20 10:06:48.612503 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.612480    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdvlr\" (UniqueName: \"kubernetes.io/projected/0e0ba9ce-b097-48e8-ad30-268f95740f34-kube-api-access-bdvlr\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst\" (UID: \"0e0ba9ce-b097-48e8-ad30-268f95740f34\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst"
Jul 20 10:06:48.643974 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.643556    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87"]
Jul 20 10:06:48.692157 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.692066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb157950-f975-43e3-8a76-0f3171443a03-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87\" (UID: \"eb157950-f975-43e3-8a76-0f3171443a03\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87"
Jul 20 10:06:48.692157 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.692152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb157950-f975-43e3-8a76-0f3171443a03-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87\" (UID: \"eb157950-f975-43e3-8a76-0f3171443a03\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87"
Jul 20 10:06:48.692423 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.692206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz84v\" (UniqueName: \"kubernetes.io/projected/eb157950-f975-43e3-8a76-0f3171443a03-kube-api-access-bz84v\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87\" (UID: \"eb157950-f975-43e3-8a76-0f3171443a03\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87"
Jul 20 10:06:48.739463 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.739427    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst"
Jul 20 10:06:48.747010 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.746981    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz"]
Jul 20 10:06:48.750834 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.750811    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz"
Jul 20 10:06:48.768860 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.768612    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz"]
Jul 20 10:06:48.793526 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.793488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb157950-f975-43e3-8a76-0f3171443a03-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87\" (UID: \"eb157950-f975-43e3-8a76-0f3171443a03\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87"
Jul 20 10:06:48.793680 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.793552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz84v\" (UniqueName: \"kubernetes.io/projected/eb157950-f975-43e3-8a76-0f3171443a03-kube-api-access-bz84v\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87\" (UID: \"eb157950-f975-43e3-8a76-0f3171443a03\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87"
Jul 20 10:06:48.793680 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.793623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb157950-f975-43e3-8a76-0f3171443a03-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87\" (UID: \"eb157950-f975-43e3-8a76-0f3171443a03\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87"
Jul 20 10:06:48.794019 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.793966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb157950-f975-43e3-8a76-0f3171443a03-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87\" (UID: \"eb157950-f975-43e3-8a76-0f3171443a03\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87"
Jul 20 10:06:48.794019 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.793981    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb157950-f975-43e3-8a76-0f3171443a03-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87\" (UID: \"eb157950-f975-43e3-8a76-0f3171443a03\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87"
Jul 20 10:06:48.808866 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.808834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz84v\" (UniqueName: \"kubernetes.io/projected/eb157950-f975-43e3-8a76-0f3171443a03-kube-api-access-bz84v\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87\" (UID: \"eb157950-f975-43e3-8a76-0f3171443a03\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87"
Jul 20 10:06:48.817366 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.817332    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw"]
Jul 20 10:06:48.821201 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.821177    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw"
Jul 20 10:06:48.835280 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.835252    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw"]
Jul 20 10:06:48.888211 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.888161    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst"]
Jul 20 10:06:48.892218 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:06:48.892187    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0ba9ce_b097_48e8_ad30_268f95740f34.slice/crio-d79e848970e4024093f61935dccd6a6bcb33895db788a4b02fa9fb49735845f0 WatchSource:0}: Error finding container d79e848970e4024093f61935dccd6a6bcb33895db788a4b02fa9fb49735845f0: Status 404 returned error can't find the container with id d79e848970e4024093f61935dccd6a6bcb33895db788a4b02fa9fb49735845f0
Jul 20 10:06:48.894061 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.894036    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hhtz\" (UniqueName: \"kubernetes.io/projected/53eeab80-75c1-4f07-ae1e-550e2d97e697-kube-api-access-4hhtz\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz\" (UID: \"53eeab80-75c1-4f07-ae1e-550e2d97e697\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz"
Jul 20 10:06:48.894130 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.894087    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53eeab80-75c1-4f07-ae1e-550e2d97e697-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz\" (UID: \"53eeab80-75c1-4f07-ae1e-550e2d97e697\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz"
Jul 20 10:06:48.894345 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.894326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lqcl\" (UniqueName: \"kubernetes.io/projected/ea68244b-0a10-4827-b783-d38b66c675f6-kube-api-access-2lqcl\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw\" (UID: \"ea68244b-0a10-4827-b783-d38b66c675f6\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw"
Jul 20 10:06:48.894442 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.894384    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea68244b-0a10-4827-b783-d38b66c675f6-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw\" (UID: \"ea68244b-0a10-4827-b783-d38b66c675f6\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw"
Jul 20 10:06:48.894442 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.894411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53eeab80-75c1-4f07-ae1e-550e2d97e697-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz\" (UID: \"53eeab80-75c1-4f07-ae1e-550e2d97e697\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz"
Jul 20 10:06:48.894542 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.894489    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea68244b-0a10-4827-b783-d38b66c675f6-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw\" (UID: \"ea68244b-0a10-4827-b783-d38b66c675f6\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw"
Jul 20 10:06:48.921099 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.921072    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87"
Jul 20 10:06:48.995556 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.995521    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lqcl\" (UniqueName: \"kubernetes.io/projected/ea68244b-0a10-4827-b783-d38b66c675f6-kube-api-access-2lqcl\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw\" (UID: \"ea68244b-0a10-4827-b783-d38b66c675f6\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw"
Jul 20 10:06:48.995748 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.995575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea68244b-0a10-4827-b783-d38b66c675f6-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw\" (UID: \"ea68244b-0a10-4827-b783-d38b66c675f6\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw"
Jul 20 10:06:48.995748 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.995610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53eeab80-75c1-4f07-ae1e-550e2d97e697-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz\" (UID: \"53eeab80-75c1-4f07-ae1e-550e2d97e697\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz"
Jul 20 10:06:48.995748 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.995658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea68244b-0a10-4827-b783-d38b66c675f6-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw\" (UID: \"ea68244b-0a10-4827-b783-d38b66c675f6\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw"
Jul 20 10:06:48.995748 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.995721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hhtz\" (UniqueName: \"kubernetes.io/projected/53eeab80-75c1-4f07-ae1e-550e2d97e697-kube-api-access-4hhtz\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz\" (UID: \"53eeab80-75c1-4f07-ae1e-550e2d97e697\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz"
Jul 20 10:06:48.995988 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.995763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53eeab80-75c1-4f07-ae1e-550e2d97e697-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz\" (UID: \"53eeab80-75c1-4f07-ae1e-550e2d97e697\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz"
Jul 20 10:06:48.996063 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.996040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea68244b-0a10-4827-b783-d38b66c675f6-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw\" (UID: \"ea68244b-0a10-4827-b783-d38b66c675f6\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw"
Jul 20 10:06:48.996122 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.996079    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53eeab80-75c1-4f07-ae1e-550e2d97e697-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz\" (UID: \"53eeab80-75c1-4f07-ae1e-550e2d97e697\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz"
Jul 20 10:06:48.996122 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.996100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53eeab80-75c1-4f07-ae1e-550e2d97e697-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz\" (UID: \"53eeab80-75c1-4f07-ae1e-550e2d97e697\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz"
Jul 20 10:06:48.996122 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:48.996110    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea68244b-0a10-4827-b783-d38b66c675f6-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw\" (UID: \"ea68244b-0a10-4827-b783-d38b66c675f6\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw"
Jul 20 10:06:49.008234 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.008210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hhtz\" (UniqueName: \"kubernetes.io/projected/53eeab80-75c1-4f07-ae1e-550e2d97e697-kube-api-access-4hhtz\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz\" (UID: \"53eeab80-75c1-4f07-ae1e-550e2d97e697\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz"
Jul 20 10:06:49.008366 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.008264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lqcl\" (UniqueName: \"kubernetes.io/projected/ea68244b-0a10-4827-b783-d38b66c675f6-kube-api-access-2lqcl\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw\" (UID: \"ea68244b-0a10-4827-b783-d38b66c675f6\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw"
Jul 20 10:06:49.064776 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.064740    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87"]
Jul 20 10:06:49.068723 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:06:49.068700    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb157950_f975_43e3_8a76_0f3171443a03.slice/crio-555b9ac4b3c4a56d2e2388b2c6723fcbbadf766164f0fdbc6a1a855194d3d3c0 WatchSource:0}: Error finding container 555b9ac4b3c4a56d2e2388b2c6723fcbbadf766164f0fdbc6a1a855194d3d3c0: Status 404 returned error can't find the container with id 555b9ac4b3c4a56d2e2388b2c6723fcbbadf766164f0fdbc6a1a855194d3d3c0
Jul 20 10:06:49.071161 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.071143    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz"
Jul 20 10:06:49.131041 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.130995    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw"
Jul 20 10:06:49.226571 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.226491    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz"]
Jul 20 10:06:49.251241 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:06:49.251203    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53eeab80_75c1_4f07_ae1e_550e2d97e697.slice/crio-9fa1fc11a55dc0f4a1b2d93f5dc9797a6cfac3ef7ae19bb1132392b83c763d1f WatchSource:0}: Error finding container 9fa1fc11a55dc0f4a1b2d93f5dc9797a6cfac3ef7ae19bb1132392b83c763d1f: Status 404 returned error can't find the container with id 9fa1fc11a55dc0f4a1b2d93f5dc9797a6cfac3ef7ae19bb1132392b83c763d1f
Jul 20 10:06:49.283067 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.282960    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw"]
Jul 20 10:06:49.285158 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:06:49.285129    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea68244b_0a10_4827_b783_d38b66c675f6.slice/crio-8642f086c70596b4437144f4a77358dfea3096a546121caed43ca54da4485c78 WatchSource:0}: Error finding container 8642f086c70596b4437144f4a77358dfea3096a546121caed43ca54da4485c78: Status 404 returned error can't find the container with id 8642f086c70596b4437144f4a77358dfea3096a546121caed43ca54da4485c78
Jul 20 10:06:49.830307 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.830269    2580 generic.go:356] "Generic (PLEG): container finished" podID="53eeab80-75c1-4f07-ae1e-550e2d97e697" containerID="7c9454388936ed9c70cdb2d56801f89c2d109df185f5c4396316227c866e2478" exitCode=0
Jul 20 10:06:49.830716 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.830353    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz" event={"ID":"53eeab80-75c1-4f07-ae1e-550e2d97e697","Type":"ContainerDied","Data":"7c9454388936ed9c70cdb2d56801f89c2d109df185f5c4396316227c866e2478"}
Jul 20 10:06:49.830716 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.830403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz" event={"ID":"53eeab80-75c1-4f07-ae1e-550e2d97e697","Type":"ContainerStarted","Data":"9fa1fc11a55dc0f4a1b2d93f5dc9797a6cfac3ef7ae19bb1132392b83c763d1f"}
Jul 20 10:06:49.832006 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.831983    2580 generic.go:356] "Generic (PLEG): container finished" podID="ea68244b-0a10-4827-b783-d38b66c675f6" containerID="ed3fb78ff10139851acd8db8a2c2c3c6b44b9d5b7e06f50314b9d05cac206b37" exitCode=0
Jul 20 10:06:49.832106 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.832025    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw" event={"ID":"ea68244b-0a10-4827-b783-d38b66c675f6","Type":"ContainerDied","Data":"ed3fb78ff10139851acd8db8a2c2c3c6b44b9d5b7e06f50314b9d05cac206b37"}
Jul 20 10:06:49.832106 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.832069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw" event={"ID":"ea68244b-0a10-4827-b783-d38b66c675f6","Type":"ContainerStarted","Data":"8642f086c70596b4437144f4a77358dfea3096a546121caed43ca54da4485c78"}
Jul 20 10:06:49.833699 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.833676    2580 generic.go:356] "Generic (PLEG): container finished" podID="0e0ba9ce-b097-48e8-ad30-268f95740f34" containerID="e132c0ddd1f7fc1dbaf2718684dff40f8b6336bd1eae03c4caa8252b1b797df4" exitCode=0
Jul 20 10:06:49.833781 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.833714    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst" event={"ID":"0e0ba9ce-b097-48e8-ad30-268f95740f34","Type":"ContainerDied","Data":"e132c0ddd1f7fc1dbaf2718684dff40f8b6336bd1eae03c4caa8252b1b797df4"}
Jul 20 10:06:49.833781 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.833753    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst" event={"ID":"0e0ba9ce-b097-48e8-ad30-268f95740f34","Type":"ContainerStarted","Data":"d79e848970e4024093f61935dccd6a6bcb33895db788a4b02fa9fb49735845f0"}
Jul 20 10:06:49.835594 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.835572    2580 generic.go:356] "Generic (PLEG): container finished" podID="eb157950-f975-43e3-8a76-0f3171443a03" containerID="3bfe7a266085d1aedfff4c43c0e0d57d206067f86ca5a102e9eebf16d8d6af64" exitCode=0
Jul 20 10:06:49.835675 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.835605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87" event={"ID":"eb157950-f975-43e3-8a76-0f3171443a03","Type":"ContainerDied","Data":"3bfe7a266085d1aedfff4c43c0e0d57d206067f86ca5a102e9eebf16d8d6af64"}
Jul 20 10:06:49.835675 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:49.835637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87" event={"ID":"eb157950-f975-43e3-8a76-0f3171443a03","Type":"ContainerStarted","Data":"555b9ac4b3c4a56d2e2388b2c6723fcbbadf766164f0fdbc6a1a855194d3d3c0"}
Jul 20 10:06:50.843742 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:50.843652    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz" event={"ID":"53eeab80-75c1-4f07-ae1e-550e2d97e697","Type":"ContainerStarted","Data":"b83b5f1dba5d66c28929b7fae1b9ac8446bf1534ed53069ad6f3736ef88b04bf"}
Jul 20 10:06:50.845982 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:50.845950    2580 generic.go:356] "Generic (PLEG): container finished" podID="ea68244b-0a10-4827-b783-d38b66c675f6" containerID="7e61c89f9fe4d5934813e2927d1731936586fe10a906a49399a135ab19c4aeb1" exitCode=0
Jul 20 10:06:50.846114 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:50.846018    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw" event={"ID":"ea68244b-0a10-4827-b783-d38b66c675f6","Type":"ContainerDied","Data":"7e61c89f9fe4d5934813e2927d1731936586fe10a906a49399a135ab19c4aeb1"}
Jul 20 10:06:51.852976 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:51.852944    2580 generic.go:356] "Generic (PLEG): container finished" podID="53eeab80-75c1-4f07-ae1e-550e2d97e697" containerID="b83b5f1dba5d66c28929b7fae1b9ac8446bf1534ed53069ad6f3736ef88b04bf" exitCode=0
Jul 20 10:06:51.853476 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:51.852993    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz" event={"ID":"53eeab80-75c1-4f07-ae1e-550e2d97e697","Type":"ContainerDied","Data":"b83b5f1dba5d66c28929b7fae1b9ac8446bf1534ed53069ad6f3736ef88b04bf"}
Jul 20 10:06:51.855699 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:51.855671    2580 generic.go:356] "Generic (PLEG): container finished" podID="ea68244b-0a10-4827-b783-d38b66c675f6" containerID="9363fc89e6938cd423509a619218f26c888041ec5729ae0ed00818721deb517c" exitCode=0
Jul 20 10:06:51.855804 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:51.855699    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw" event={"ID":"ea68244b-0a10-4827-b783-d38b66c675f6","Type":"ContainerDied","Data":"9363fc89e6938cd423509a619218f26c888041ec5729ae0ed00818721deb517c"}
Jul 20 10:06:51.857755 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:51.857728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst" event={"ID":"0e0ba9ce-b097-48e8-ad30-268f95740f34","Type":"ContainerStarted","Data":"99310b9e38eca6dac570c758fbd87db5441bc751be0ed51f504c2835cf882a6e"}
Jul 20 10:06:51.860000 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:51.859930    2580 generic.go:356] "Generic (PLEG): container finished" podID="eb157950-f975-43e3-8a76-0f3171443a03" containerID="8899f0e6d4548ec2698936c927132eca86e2fd2edc820b9d66e2987157dbba89" exitCode=0
Jul 20 10:06:51.860000 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:51.859976    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87" event={"ID":"eb157950-f975-43e3-8a76-0f3171443a03","Type":"ContainerDied","Data":"8899f0e6d4548ec2698936c927132eca86e2fd2edc820b9d66e2987157dbba89"}
Jul 20 10:06:52.872298 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:52.872259    2580 generic.go:356] "Generic (PLEG): container finished" podID="eb157950-f975-43e3-8a76-0f3171443a03" containerID="9870e90b3e947bdcef4270dd9c3627454770c42e0c7f0b723d786e6f674280e4" exitCode=0
Jul 20 10:06:52.872846 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:52.872339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87" event={"ID":"eb157950-f975-43e3-8a76-0f3171443a03","Type":"ContainerDied","Data":"9870e90b3e947bdcef4270dd9c3627454770c42e0c7f0b723d786e6f674280e4"}
Jul 20 10:06:52.874353 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:52.874329    2580 generic.go:356] "Generic (PLEG): container finished" podID="53eeab80-75c1-4f07-ae1e-550e2d97e697" containerID="b441a809d8d52f9c27c3aa19ce1dd905eab28e45fb69dffad1b2896071df7eae" exitCode=0
Jul 20 10:06:52.874525 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:52.874421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz" event={"ID":"53eeab80-75c1-4f07-ae1e-550e2d97e697","Type":"ContainerDied","Data":"b441a809d8d52f9c27c3aa19ce1dd905eab28e45fb69dffad1b2896071df7eae"}
Jul 20 10:06:52.876038 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:52.876014    2580 generic.go:356] "Generic (PLEG): container finished" podID="0e0ba9ce-b097-48e8-ad30-268f95740f34" containerID="99310b9e38eca6dac570c758fbd87db5441bc751be0ed51f504c2835cf882a6e" exitCode=0
Jul 20 10:06:52.876147 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:52.876088    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst" event={"ID":"0e0ba9ce-b097-48e8-ad30-268f95740f34","Type":"ContainerDied","Data":"99310b9e38eca6dac570c758fbd87db5441bc751be0ed51f504c2835cf882a6e"}
Jul 20 10:06:53.018974 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:53.018950    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw"
Jul 20 10:06:53.132312 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:53.132279    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lqcl\" (UniqueName: \"kubernetes.io/projected/ea68244b-0a10-4827-b783-d38b66c675f6-kube-api-access-2lqcl\") pod \"ea68244b-0a10-4827-b783-d38b66c675f6\" (UID: \"ea68244b-0a10-4827-b783-d38b66c675f6\") "
Jul 20 10:06:53.132511 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:53.132368    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea68244b-0a10-4827-b783-d38b66c675f6-util\") pod \"ea68244b-0a10-4827-b783-d38b66c675f6\" (UID: \"ea68244b-0a10-4827-b783-d38b66c675f6\") "
Jul 20 10:06:53.132511 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:53.132394    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea68244b-0a10-4827-b783-d38b66c675f6-bundle\") pod \"ea68244b-0a10-4827-b783-d38b66c675f6\" (UID: \"ea68244b-0a10-4827-b783-d38b66c675f6\") "
Jul 20 10:06:53.132880 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:53.132857    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea68244b-0a10-4827-b783-d38b66c675f6-bundle" (OuterVolumeSpecName: "bundle") pod "ea68244b-0a10-4827-b783-d38b66c675f6" (UID: "ea68244b-0a10-4827-b783-d38b66c675f6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:06:53.134588 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:53.134563    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea68244b-0a10-4827-b783-d38b66c675f6-kube-api-access-2lqcl" (OuterVolumeSpecName: "kube-api-access-2lqcl") pod "ea68244b-0a10-4827-b783-d38b66c675f6" (UID: "ea68244b-0a10-4827-b783-d38b66c675f6"). InnerVolumeSpecName "kube-api-access-2lqcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:06:53.137588 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:53.137563    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea68244b-0a10-4827-b783-d38b66c675f6-util" (OuterVolumeSpecName: "util") pod "ea68244b-0a10-4827-b783-d38b66c675f6" (UID: "ea68244b-0a10-4827-b783-d38b66c675f6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:06:53.233366 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:53.233329    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea68244b-0a10-4827-b783-d38b66c675f6-util\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:06:53.233366 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:53.233363    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea68244b-0a10-4827-b783-d38b66c675f6-bundle\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:06:53.233366 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:53.233373    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lqcl\" (UniqueName: \"kubernetes.io/projected/ea68244b-0a10-4827-b783-d38b66c675f6-kube-api-access-2lqcl\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:06:53.882462 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:53.882423    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw"
Jul 20 10:06:53.882874 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:53.882424    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabpmrjw" event={"ID":"ea68244b-0a10-4827-b783-d38b66c675f6","Type":"ContainerDied","Data":"8642f086c70596b4437144f4a77358dfea3096a546121caed43ca54da4485c78"}
Jul 20 10:06:53.882874 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:53.882558    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8642f086c70596b4437144f4a77358dfea3096a546121caed43ca54da4485c78"
Jul 20 10:06:53.884620 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:53.884598    2580 generic.go:356] "Generic (PLEG): container finished" podID="0e0ba9ce-b097-48e8-ad30-268f95740f34" containerID="3f48eb162533eb870538c84a8037e8ce4c6ff60e61e7f92a22c567832215fab8" exitCode=0
Jul 20 10:06:53.884750 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:53.884671    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst" event={"ID":"0e0ba9ce-b097-48e8-ad30-268f95740f34","Type":"ContainerDied","Data":"3f48eb162533eb870538c84a8037e8ce4c6ff60e61e7f92a22c567832215fab8"}
Jul 20 10:06:54.021524 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.021498    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87"
Jul 20 10:06:54.049210 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.049160    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz"
Jul 20 10:06:54.142230 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.142192    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53eeab80-75c1-4f07-ae1e-550e2d97e697-bundle\") pod \"53eeab80-75c1-4f07-ae1e-550e2d97e697\" (UID: \"53eeab80-75c1-4f07-ae1e-550e2d97e697\") "
Jul 20 10:06:54.142489 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.142247    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb157950-f975-43e3-8a76-0f3171443a03-util\") pod \"eb157950-f975-43e3-8a76-0f3171443a03\" (UID: \"eb157950-f975-43e3-8a76-0f3171443a03\") "
Jul 20 10:06:54.142489 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.142281    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hhtz\" (UniqueName: \"kubernetes.io/projected/53eeab80-75c1-4f07-ae1e-550e2d97e697-kube-api-access-4hhtz\") pod \"53eeab80-75c1-4f07-ae1e-550e2d97e697\" (UID: \"53eeab80-75c1-4f07-ae1e-550e2d97e697\") "
Jul 20 10:06:54.142489 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.142315    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb157950-f975-43e3-8a76-0f3171443a03-bundle\") pod \"eb157950-f975-43e3-8a76-0f3171443a03\" (UID: \"eb157950-f975-43e3-8a76-0f3171443a03\") "
Jul 20 10:06:54.142489 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.142374    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53eeab80-75c1-4f07-ae1e-550e2d97e697-util\") pod \"53eeab80-75c1-4f07-ae1e-550e2d97e697\" (UID: \"53eeab80-75c1-4f07-ae1e-550e2d97e697\") "
Jul 20 10:06:54.142489 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.142432    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bz84v\" (UniqueName: \"kubernetes.io/projected/eb157950-f975-43e3-8a76-0f3171443a03-kube-api-access-bz84v\") pod \"eb157950-f975-43e3-8a76-0f3171443a03\" (UID: \"eb157950-f975-43e3-8a76-0f3171443a03\") "
Jul 20 10:06:54.142722 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.142698    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53eeab80-75c1-4f07-ae1e-550e2d97e697-bundle" (OuterVolumeSpecName: "bundle") pod "53eeab80-75c1-4f07-ae1e-550e2d97e697" (UID: "53eeab80-75c1-4f07-ae1e-550e2d97e697"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:06:54.142893 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.142866    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb157950-f975-43e3-8a76-0f3171443a03-bundle" (OuterVolumeSpecName: "bundle") pod "eb157950-f975-43e3-8a76-0f3171443a03" (UID: "eb157950-f975-43e3-8a76-0f3171443a03"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:06:54.143001 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.142981    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53eeab80-75c1-4f07-ae1e-550e2d97e697-bundle\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:06:54.144815 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.144785    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb157950-f975-43e3-8a76-0f3171443a03-kube-api-access-bz84v" (OuterVolumeSpecName: "kube-api-access-bz84v") pod "eb157950-f975-43e3-8a76-0f3171443a03" (UID: "eb157950-f975-43e3-8a76-0f3171443a03"). InnerVolumeSpecName "kube-api-access-bz84v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:06:54.144988 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.144968    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53eeab80-75c1-4f07-ae1e-550e2d97e697-kube-api-access-4hhtz" (OuterVolumeSpecName: "kube-api-access-4hhtz") pod "53eeab80-75c1-4f07-ae1e-550e2d97e697" (UID: "53eeab80-75c1-4f07-ae1e-550e2d97e697"). InnerVolumeSpecName "kube-api-access-4hhtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:06:54.148238 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.148213    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53eeab80-75c1-4f07-ae1e-550e2d97e697-util" (OuterVolumeSpecName: "util") pod "53eeab80-75c1-4f07-ae1e-550e2d97e697" (UID: "53eeab80-75c1-4f07-ae1e-550e2d97e697"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:06:54.148868 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.148850    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb157950-f975-43e3-8a76-0f3171443a03-util" (OuterVolumeSpecName: "util") pod "eb157950-f975-43e3-8a76-0f3171443a03" (UID: "eb157950-f975-43e3-8a76-0f3171443a03"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:06:54.243740 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.243702    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb157950-f975-43e3-8a76-0f3171443a03-util\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:06:54.243740 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.243735    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hhtz\" (UniqueName: \"kubernetes.io/projected/53eeab80-75c1-4f07-ae1e-550e2d97e697-kube-api-access-4hhtz\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:06:54.243740 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.243747    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb157950-f975-43e3-8a76-0f3171443a03-bundle\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:06:54.243982 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.243756    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53eeab80-75c1-4f07-ae1e-550e2d97e697-util\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:06:54.243982 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.243764    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz84v\" (UniqueName: \"kubernetes.io/projected/eb157950-f975-43e3-8a76-0f3171443a03-kube-api-access-bz84v\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:06:54.891826 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.891793    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87"
Jul 20 10:06:54.892292 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.891793    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36q4x87" event={"ID":"eb157950-f975-43e3-8a76-0f3171443a03","Type":"ContainerDied","Data":"555b9ac4b3c4a56d2e2388b2c6723fcbbadf766164f0fdbc6a1a855194d3d3c0"}
Jul 20 10:06:54.892292 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.891942    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="555b9ac4b3c4a56d2e2388b2c6723fcbbadf766164f0fdbc6a1a855194d3d3c0"
Jul 20 10:06:54.893867 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.893840    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz"
Jul 20 10:06:54.893867 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.893862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efj2zrz" event={"ID":"53eeab80-75c1-4f07-ae1e-550e2d97e697","Type":"ContainerDied","Data":"9fa1fc11a55dc0f4a1b2d93f5dc9797a6cfac3ef7ae19bb1132392b83c763d1f"}
Jul 20 10:06:54.894060 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:54.893901    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fa1fc11a55dc0f4a1b2d93f5dc9797a6cfac3ef7ae19bb1132392b83c763d1f"
Jul 20 10:06:55.026326 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:55.026303    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst"
Jul 20 10:06:55.152438 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:55.152402    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdvlr\" (UniqueName: \"kubernetes.io/projected/0e0ba9ce-b097-48e8-ad30-268f95740f34-kube-api-access-bdvlr\") pod \"0e0ba9ce-b097-48e8-ad30-268f95740f34\" (UID: \"0e0ba9ce-b097-48e8-ad30-268f95740f34\") "
Jul 20 10:06:55.152645 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:55.152480    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e0ba9ce-b097-48e8-ad30-268f95740f34-util\") pod \"0e0ba9ce-b097-48e8-ad30-268f95740f34\" (UID: \"0e0ba9ce-b097-48e8-ad30-268f95740f34\") "
Jul 20 10:06:55.152645 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:55.152516    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e0ba9ce-b097-48e8-ad30-268f95740f34-bundle\") pod \"0e0ba9ce-b097-48e8-ad30-268f95740f34\" (UID: \"0e0ba9ce-b097-48e8-ad30-268f95740f34\") "
Jul 20 10:06:55.153319 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:55.153290    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0ba9ce-b097-48e8-ad30-268f95740f34-bundle" (OuterVolumeSpecName: "bundle") pod "0e0ba9ce-b097-48e8-ad30-268f95740f34" (UID: "0e0ba9ce-b097-48e8-ad30-268f95740f34"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:06:55.154684 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:55.154664    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0ba9ce-b097-48e8-ad30-268f95740f34-kube-api-access-bdvlr" (OuterVolumeSpecName: "kube-api-access-bdvlr") pod "0e0ba9ce-b097-48e8-ad30-268f95740f34" (UID: "0e0ba9ce-b097-48e8-ad30-268f95740f34"). InnerVolumeSpecName "kube-api-access-bdvlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:06:55.156624 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:55.156593    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0ba9ce-b097-48e8-ad30-268f95740f34-util" (OuterVolumeSpecName: "util") pod "0e0ba9ce-b097-48e8-ad30-268f95740f34" (UID: "0e0ba9ce-b097-48e8-ad30-268f95740f34"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:06:55.253910 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:55.253879    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e0ba9ce-b097-48e8-ad30-268f95740f34-util\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:06:55.253910 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:55.253905    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e0ba9ce-b097-48e8-ad30-268f95740f34-bundle\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:06:55.253910 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:55.253914    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdvlr\" (UniqueName: \"kubernetes.io/projected/0e0ba9ce-b097-48e8-ad30-268f95740f34-kube-api-access-bdvlr\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:06:55.900559 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:55.900515    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst" event={"ID":"0e0ba9ce-b097-48e8-ad30-268f95740f34","Type":"ContainerDied","Data":"d79e848970e4024093f61935dccd6a6bcb33895db788a4b02fa9fb49735845f0"}
Jul 20 10:06:55.900559 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:55.900559    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76g8cst"
Jul 20 10:06:55.900975 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:55.900572    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d79e848970e4024093f61935dccd6a6bcb33895db788a4b02fa9fb49735845f0"
Jul 20 10:06:58.248766 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:06:58.248730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:01.059468 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:01.059431    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:02.410833 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.410803    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-rvhvm"]
Jul 20 10:07:02.411241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411097    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53eeab80-75c1-4f07-ae1e-550e2d97e697" containerName="pull"
Jul 20 10:07:02.411241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411107    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53eeab80-75c1-4f07-ae1e-550e2d97e697" containerName="pull"
Jul 20 10:07:02.411241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411116    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea68244b-0a10-4827-b783-d38b66c675f6" containerName="util"
Jul 20 10:07:02.411241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411121    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea68244b-0a10-4827-b783-d38b66c675f6" containerName="util"
Jul 20 10:07:02.411241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411130    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea68244b-0a10-4827-b783-d38b66c675f6" containerName="pull"
Jul 20 10:07:02.411241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411137    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea68244b-0a10-4827-b783-d38b66c675f6" containerName="pull"
Jul 20 10:07:02.411241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411149    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e0ba9ce-b097-48e8-ad30-268f95740f34" containerName="pull"
Jul 20 10:07:02.411241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411154    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e0ba9ce-b097-48e8-ad30-268f95740f34" containerName="pull"
Jul 20 10:07:02.411241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411161    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb157950-f975-43e3-8a76-0f3171443a03" containerName="pull"
Jul 20 10:07:02.411241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411187    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb157950-f975-43e3-8a76-0f3171443a03" containerName="pull"
Jul 20 10:07:02.411241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411196    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e0ba9ce-b097-48e8-ad30-268f95740f34" containerName="util"
Jul 20 10:07:02.411241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411201    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e0ba9ce-b097-48e8-ad30-268f95740f34" containerName="util"
Jul 20 10:07:02.411241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411209    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53eeab80-75c1-4f07-ae1e-550e2d97e697" containerName="extract"
Jul 20 10:07:02.411241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411215    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53eeab80-75c1-4f07-ae1e-550e2d97e697" containerName="extract"
Jul 20 10:07:02.411241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411227    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea68244b-0a10-4827-b783-d38b66c675f6" containerName="extract"
Jul 20 10:07:02.411241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411234    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea68244b-0a10-4827-b783-d38b66c675f6" containerName="extract"
Jul 20 10:07:02.411241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411241    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb157950-f975-43e3-8a76-0f3171443a03" containerName="util"
Jul 20 10:07:02.411241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411246    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb157950-f975-43e3-8a76-0f3171443a03" containerName="util"
Jul 20 10:07:02.411746 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411256    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb157950-f975-43e3-8a76-0f3171443a03" containerName="extract"
Jul 20 10:07:02.411746 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411261    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb157950-f975-43e3-8a76-0f3171443a03" containerName="extract"
Jul 20 10:07:02.411746 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411265    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53eeab80-75c1-4f07-ae1e-550e2d97e697" containerName="util"
Jul 20 10:07:02.411746 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411270    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53eeab80-75c1-4f07-ae1e-550e2d97e697" containerName="util"
Jul 20 10:07:02.411746 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411278    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e0ba9ce-b097-48e8-ad30-268f95740f34" containerName="extract"
Jul 20 10:07:02.411746 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411283    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e0ba9ce-b097-48e8-ad30-268f95740f34" containerName="extract"
Jul 20 10:07:02.411746 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411339    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb157950-f975-43e3-8a76-0f3171443a03" containerName="extract"
Jul 20 10:07:02.411746 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411347    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea68244b-0a10-4827-b783-d38b66c675f6" containerName="extract"
Jul 20 10:07:02.411746 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411353    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="53eeab80-75c1-4f07-ae1e-550e2d97e697" containerName="extract"
Jul 20 10:07:02.411746 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.411359    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e0ba9ce-b097-48e8-ad30-268f95740f34" containerName="extract"
Jul 20 10:07:02.414914 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.414896    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rvhvm"
Jul 20 10:07:02.419354 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.419332    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 20 10:07:02.419839 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.419822    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-66n42\""
Jul 20 10:07:02.419924 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.419863    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 20 10:07:02.431036 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.431008    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-rvhvm"]
Jul 20 10:07:02.518459 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.518424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l4kv\" (UniqueName: \"kubernetes.io/projected/95ce9651-daa6-4848-aedb-dd91397e2d6f-kube-api-access-9l4kv\") pod \"limitador-operator-controller-manager-6f9f468797-rvhvm\" (UID: \"95ce9651-daa6-4848-aedb-dd91397e2d6f\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rvhvm"
Jul 20 10:07:02.619480 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.619438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l4kv\" (UniqueName: \"kubernetes.io/projected/95ce9651-daa6-4848-aedb-dd91397e2d6f-kube-api-access-9l4kv\") pod \"limitador-operator-controller-manager-6f9f468797-rvhvm\" (UID: \"95ce9651-daa6-4848-aedb-dd91397e2d6f\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rvhvm"
Jul 20 10:07:02.635206 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.635159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l4kv\" (UniqueName: \"kubernetes.io/projected/95ce9651-daa6-4848-aedb-dd91397e2d6f-kube-api-access-9l4kv\") pod \"limitador-operator-controller-manager-6f9f468797-rvhvm\" (UID: \"95ce9651-daa6-4848-aedb-dd91397e2d6f\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rvhvm"
Jul 20 10:07:02.727736 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.727697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rvhvm"
Jul 20 10:07:02.879505 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.879482    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-rvhvm"]
Jul 20 10:07:02.881379 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:07:02.881351    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ce9651_daa6_4848_aedb_dd91397e2d6f.slice/crio-5dbc6902d26f254f3be22f2fa9cea100c0724d6b909cd59c90322e6df53dccfd WatchSource:0}: Error finding container 5dbc6902d26f254f3be22f2fa9cea100c0724d6b909cd59c90322e6df53dccfd: Status 404 returned error can't find the container with id 5dbc6902d26f254f3be22f2fa9cea100c0724d6b909cd59c90322e6df53dccfd
Jul 20 10:07:02.938658 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:02.938619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rvhvm" event={"ID":"95ce9651-daa6-4848-aedb-dd91397e2d6f","Type":"ContainerStarted","Data":"5dbc6902d26f254f3be22f2fa9cea100c0724d6b909cd59c90322e6df53dccfd"}
Jul 20 10:07:03.007845 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:03.007751    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:03.110156 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:03.110109    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:03.641754 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:03.641711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:04.022523 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:04.022492    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:04.340056 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:04.339971    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:04.953533 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:04.953503    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rvhvm" event={"ID":"95ce9651-daa6-4848-aedb-dd91397e2d6f","Type":"ContainerStarted","Data":"c0ee32db5dea03aa75d121e6dd0571f68ccea5fa0362c2070270630d5331ef04"}
Jul 20 10:07:04.953938 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:04.953575    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rvhvm"
Jul 20 10:07:04.975908 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:04.975859    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rvhvm" podStartSLOduration=1.315512137 podStartE2EDuration="2.975841105s" totalImagesPullingTime="1.660328968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:07:04.973066648 +0000 UTC m=+637.021930294" watchObservedRunningTime="2026-07-20 10:07:04.975841105 +0000 UTC m=+637.024704752"
Jul 20 10:07:05.114607 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:05.114571    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:05.215548 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:05.215462    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:07.236978 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:07.236940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:07.624954 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:07.624909    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:09.103723 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:09.103682    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:09.228843 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:09.228811    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:09.614423 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:09.614388    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-kxw42"]
Jul 20 10:07:09.617497 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:09.617481    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6d85f6564-kxw42"
Jul 20 10:07:09.620182 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:09.620148    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-s49t5\""
Jul 20 10:07:09.635792 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:09.635765    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-kxw42"]
Jul 20 10:07:09.783670 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:09.783627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbxtp\" (UniqueName: \"kubernetes.io/projected/76ff9bc1-31e3-4adf-b705-42b53b9c122c-kube-api-access-bbxtp\") pod \"authorino-operator-6d85f6564-kxw42\" (UID: \"76ff9bc1-31e3-4adf-b705-42b53b9c122c\") " pod="kuadrant-system/authorino-operator-6d85f6564-kxw42"
Jul 20 10:07:09.885069 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:09.884973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbxtp\" (UniqueName: \"kubernetes.io/projected/76ff9bc1-31e3-4adf-b705-42b53b9c122c-kube-api-access-bbxtp\") pod \"authorino-operator-6d85f6564-kxw42\" (UID: \"76ff9bc1-31e3-4adf-b705-42b53b9c122c\") " pod="kuadrant-system/authorino-operator-6d85f6564-kxw42"
Jul 20 10:07:09.905787 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:09.905762    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbxtp\" (UniqueName: \"kubernetes.io/projected/76ff9bc1-31e3-4adf-b705-42b53b9c122c-kube-api-access-bbxtp\") pod \"authorino-operator-6d85f6564-kxw42\" (UID: \"76ff9bc1-31e3-4adf-b705-42b53b9c122c\") " pod="kuadrant-system/authorino-operator-6d85f6564-kxw42"
Jul 20 10:07:09.927585 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:09.927552    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6d85f6564-kxw42"
Jul 20 10:07:10.071349 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:10.071323    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-kxw42"]
Jul 20 10:07:10.073496 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:07:10.073469    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ff9bc1_31e3_4adf_b705_42b53b9c122c.slice/crio-34b6779af3ed22ab20696f0050d324227dbc2a1f52cb9b503f4cd88f37952992 WatchSource:0}: Error finding container 34b6779af3ed22ab20696f0050d324227dbc2a1f52cb9b503f4cd88f37952992: Status 404 returned error can't find the container with id 34b6779af3ed22ab20696f0050d324227dbc2a1f52cb9b503f4cd88f37952992
Jul 20 10:07:10.986947 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:10.986909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6d85f6564-kxw42" event={"ID":"76ff9bc1-31e3-4adf-b705-42b53b9c122c","Type":"ContainerStarted","Data":"34b6779af3ed22ab20696f0050d324227dbc2a1f52cb9b503f4cd88f37952992"}
Jul 20 10:07:11.069746 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:11.069663    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:11.179539 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:11.179488    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:13.001139 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:13.001102    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6d85f6564-kxw42" event={"ID":"76ff9bc1-31e3-4adf-b705-42b53b9c122c","Type":"ContainerStarted","Data":"623b57d369ad1934420156cb569e53590dfdb5c3e025faacb6df155225bfcdcc"}
Jul 20 10:07:13.001662 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:13.001219    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6d85f6564-kxw42"
Jul 20 10:07:13.019480 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:13.019429    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6d85f6564-kxw42" podStartSLOduration=1.644799378 podStartE2EDuration="4.019412426s" totalImagesPullingTime="2.374613048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:07:13.017708535 +0000 UTC m=+645.066572182" watchObservedRunningTime="2026-07-20 10:07:13.019412426 +0000 UTC m=+645.068276073"
Jul 20 10:07:13.921996 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:13.921943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:14.325827 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:14.325786    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:14.619719 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:14.619622    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:14.721625 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:14.721596    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:15.050675 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:15.050638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:15.115926 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:15.115887    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:15.961044 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:15.961016    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rvhvm"
Jul 20 10:07:18.782663 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:18.782626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:18.881318 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:18.881286    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:18.981935 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:18.981899    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:24.009440 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:24.009407    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6d85f6564-kxw42"
Jul 20 10:07:27.636260 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:27.636218    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:27.733687 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:27.733644    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:27.808997 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:27.808967    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:30.224635 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:30.224594    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:30.326081 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:30.326043    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:33.435772 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:33.435737    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:33.537770 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:33.537721    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:37.876945 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:37.876909    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:38.084731 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:38.084697    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:38.190969 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:38.190877    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:38.291158 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:38.291118    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:46.578376 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:46.578346    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:46.686290 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:46.686245    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:46.783881 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:46.783838    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:46.885081 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:46.885004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:54.836183 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:54.836126    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:56.388091 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:56.388066    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:57.863481 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:57.863439    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:57.970608 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:57.970568    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:58.049056 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:58.049025    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:58.151995 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:58.151913    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-q97v8"]
Jul 20 10:07:58.154482 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:58.154462    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-q97v8"
Jul 20 10:07:58.156989 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:58.156952    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 20 10:07:58.157596 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:58.157581    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-krh9b\""
Jul 20 10:07:58.167604 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:58.167578    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-q97v8"]
Jul 20 10:07:58.208098 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:58.208062    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-q97v8"]
Jul 20 10:07:58.276907 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:58.276869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5c67341c-9eb9-4b65-9661-a5dd48e6ca74-config-file\") pod \"limitador-limitador-69574b596d-q97v8\" (UID: \"5c67341c-9eb9-4b65-9661-a5dd48e6ca74\") " pod="kuadrant-system/limitador-limitador-69574b596d-q97v8"
Jul 20 10:07:58.277125 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:58.277015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tp2t\" (UniqueName: \"kubernetes.io/projected/5c67341c-9eb9-4b65-9661-a5dd48e6ca74-kube-api-access-4tp2t\") pod \"limitador-limitador-69574b596d-q97v8\" (UID: \"5c67341c-9eb9-4b65-9661-a5dd48e6ca74\") " pod="kuadrant-system/limitador-limitador-69574b596d-q97v8"
Jul 20 10:07:58.378253 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:58.378192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tp2t\" (UniqueName: \"kubernetes.io/projected/5c67341c-9eb9-4b65-9661-a5dd48e6ca74-kube-api-access-4tp2t\") pod \"limitador-limitador-69574b596d-q97v8\" (UID: \"5c67341c-9eb9-4b65-9661-a5dd48e6ca74\") " pod="kuadrant-system/limitador-limitador-69574b596d-q97v8"
Jul 20 10:07:58.378449 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:58.378280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5c67341c-9eb9-4b65-9661-a5dd48e6ca74-config-file\") pod \"limitador-limitador-69574b596d-q97v8\" (UID: \"5c67341c-9eb9-4b65-9661-a5dd48e6ca74\") " pod="kuadrant-system/limitador-limitador-69574b596d-q97v8"
Jul 20 10:07:58.378942 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:58.378920    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5c67341c-9eb9-4b65-9661-a5dd48e6ca74-config-file\") pod \"limitador-limitador-69574b596d-q97v8\" (UID: \"5c67341c-9eb9-4b65-9661-a5dd48e6ca74\") " pod="kuadrant-system/limitador-limitador-69574b596d-q97v8"
Jul 20 10:07:58.386558 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:58.386523    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tp2t\" (UniqueName: \"kubernetes.io/projected/5c67341c-9eb9-4b65-9661-a5dd48e6ca74-kube-api-access-4tp2t\") pod \"limitador-limitador-69574b596d-q97v8\" (UID: \"5c67341c-9eb9-4b65-9661-a5dd48e6ca74\") " pod="kuadrant-system/limitador-limitador-69574b596d-q97v8"
Jul 20 10:07:58.464981 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:58.464941    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-q97v8"
Jul 20 10:07:58.602729 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:58.602630    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-q97v8"]
Jul 20 10:07:58.605238 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:07:58.605212    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c67341c_9eb9_4b65_9661_a5dd48e6ca74.slice/crio-481aedf309c136c5b8e3072d0a564c92a70ac60433c849dd757f02ca491d3f09 WatchSource:0}: Error finding container 481aedf309c136c5b8e3072d0a564c92a70ac60433c849dd757f02ca491d3f09: Status 404 returned error can't find the container with id 481aedf309c136c5b8e3072d0a564c92a70ac60433c849dd757f02ca491d3f09
Jul 20 10:07:58.607782 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:58.607768    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:07:58.717581 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:58.717507    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:58.822630 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:58.822593    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:59.236908 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:07:59.236877    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-q97v8" event={"ID":"5c67341c-9eb9-4b65-9661-a5dd48e6ca74","Type":"ContainerStarted","Data":"481aedf309c136c5b8e3072d0a564c92a70ac60433c849dd757f02ca491d3f09"}
Jul 20 10:08:02.263654 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:08:02.263566    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-q97v8" event={"ID":"5c67341c-9eb9-4b65-9661-a5dd48e6ca74","Type":"ContainerStarted","Data":"62e758b56d75633ee60d7eecc4cb0858ceaef7d74aaeca50d6671ba3e1db9ab3"}
Jul 20 10:08:02.263654 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:08:02.263627    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-69574b596d-q97v8"
Jul 20 10:08:02.282776 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:08:02.282717    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-69574b596d-q97v8" podStartSLOduration=0.888945437 podStartE2EDuration="4.282698587s" totalImagesPullingTime="3.39375315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:08:02.280657766 +0000 UTC m=+694.329521436" watchObservedRunningTime="2026-07-20 10:08:02.282698587 +0000 UTC m=+694.331562234"
Jul 20 10:08:13.269666 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:08:13.269634    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-69574b596d-q97v8"
Jul 20 10:08:19.442494 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:08:19.442394    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:08:19.593236 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:08:19.593200    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:08:29.692670 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:08:29.692631    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:08:32.210021 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:08:32.209952    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:08:32.325069 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:08:32.325028    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:08:48.085932 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:08:48.085894    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:08:48.152055 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:08:48.152017    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:09:11.383018 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:09:11.382984    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:09:11.493153 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:09:11.493121    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:09:11.599737 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:09:11.599708    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:09:11.750105 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:09:11.750047    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:09:12.379472 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:09:12.379432    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:09:12.479139 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:09:12.479078    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:09:25.955527 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:09:25.955492    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:09:26.010451 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:09:26.010415    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:10:46.136576 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:10:46.136527    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:10:46.279098 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:10:46.279062    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:00.676361 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:00.676327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:00.895522 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:00.895481    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:11.251921 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:11.251887    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:11.637948 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:11.637861    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:21.264028 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:21.263939    2580 generic.go:356] "Generic (PLEG): container finished" podID="06712755-7e3d-4b4c-9c07-3260be21366b" containerID="f33529118c115cf2168c6ab7707cd0876fece2e16039240d3a010c8afd844a49" exitCode=0
Jul 20 10:11:21.264028 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:21.264001    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp" event={"ID":"06712755-7e3d-4b4c-9c07-3260be21366b","Type":"ContainerDied","Data":"f33529118c115cf2168c6ab7707cd0876fece2e16039240d3a010c8afd844a49"}
Jul 20 10:11:21.264451 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:21.264362    2580 scope.go:117] "RemoveContainer" containerID="f33529118c115cf2168c6ab7707cd0876fece2e16039240d3a010c8afd844a49"
Jul 20 10:11:21.265273 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:21.265247    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 20 10:11:21.265343 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:21.265313    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 20 10:11:21.265380 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:21.265345    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 20 10:11:22.271632 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:22.271593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp" event={"ID":"06712755-7e3d-4b4c-9c07-3260be21366b","Type":"ContainerStarted","Data":"709d441b2b4a0cf5033e90fea14b6b8bf8b9b73bebc8a30385559bb22e2d4504"}
Jul 20 10:11:22.712947 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:22.712909    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:11:22.713372 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:22.713348    2580 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Jul 20 10:11:22.713566 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:22.713538    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp" podUID="06712755-7e3d-4b4c-9c07-3260be21366b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Jul 20 10:11:22.907683 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:22.907649    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"]
Jul 20 10:11:22.911057 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:22.911035    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:22.914360 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:22.914340    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-jcgp7\""
Jul 20 10:11:22.914875 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:22.914858    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 20 10:11:22.915391 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:22.915362    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 20 10:11:22.916251 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:22.916235    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 20 10:11:22.926722 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:22.926694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-dfmzc\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:22.926904 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:22.926883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-dshm\") pod \"fc-smoke-test-kserve-7f4687897-dfmzc\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:22.927029 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:22.927012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-home\") pod \"fc-smoke-test-kserve-7f4687897-dfmzc\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:22.927206 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:22.927187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8jjz\" (UniqueName: \"kubernetes.io/projected/d67f084e-26e4-41d8-9620-1389b2cdf127-kube-api-access-g8jjz\") pod \"fc-smoke-test-kserve-7f4687897-dfmzc\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:22.927362 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:22.927347    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d67f084e-26e4-41d8-9620-1389b2cdf127-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-dfmzc\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:22.927471 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:22.927458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-dfmzc\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:22.927639 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:22.927606    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"]
Jul 20 10:11:23.012251 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:23.010005    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:23.028213 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:23.028144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-dfmzc\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:23.028213 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:23.028217    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-dshm\") pod \"fc-smoke-test-kserve-7f4687897-dfmzc\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:23.028479 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:23.028246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-home\") pod \"fc-smoke-test-kserve-7f4687897-dfmzc\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:23.028479 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:23.028311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8jjz\" (UniqueName: \"kubernetes.io/projected/d67f084e-26e4-41d8-9620-1389b2cdf127-kube-api-access-g8jjz\") pod \"fc-smoke-test-kserve-7f4687897-dfmzc\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:23.028479 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:23.028344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d67f084e-26e4-41d8-9620-1389b2cdf127-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-dfmzc\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:23.028479 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:23.028373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-dfmzc\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:23.028749 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:23.028725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-dfmzc\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:23.028819 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:23.028747    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-home\") pod \"fc-smoke-test-kserve-7f4687897-dfmzc\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:23.028917 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:23.028877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-dfmzc\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:23.030918 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:23.030888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-dshm\") pod \"fc-smoke-test-kserve-7f4687897-dfmzc\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:23.031415 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:23.031395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d67f084e-26e4-41d8-9620-1389b2cdf127-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-dfmzc\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:23.041583 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:23.041547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8jjz\" (UniqueName: \"kubernetes.io/projected/d67f084e-26e4-41d8-9620-1389b2cdf127-kube-api-access-g8jjz\") pod \"fc-smoke-test-kserve-7f4687897-dfmzc\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:23.229001 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:23.228961    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:23.308079 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:23.308013    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:23.397333 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:23.397306    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"]
Jul 20 10:11:23.399184 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:11:23.399128    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67f084e_26e4_41d8_9620_1389b2cdf127.slice/crio-abb3bb1686c34f1e407cb7592e1158be226918072cf34ee955c1b0a59bce7734 WatchSource:0}: Error finding container abb3bb1686c34f1e407cb7592e1158be226918072cf34ee955c1b0a59bce7734: Status 404 returned error can't find the container with id abb3bb1686c34f1e407cb7592e1158be226918072cf34ee955c1b0a59bce7734
Jul 20 10:11:23.718261 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:23.711602    2580 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Jul 20 10:11:23.718261 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:23.711664    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp" podUID="06712755-7e3d-4b4c-9c07-3260be21366b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Jul 20 10:11:24.285006 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:24.284928    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc" event={"ID":"d67f084e-26e4-41d8-9620-1389b2cdf127","Type":"ContainerStarted","Data":"abb3bb1686c34f1e407cb7592e1158be226918072cf34ee955c1b0a59bce7734"}
Jul 20 10:11:24.719145 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:24.718705    2580 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Jul 20 10:11:24.719145 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:24.718770    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp" podUID="06712755-7e3d-4b4c-9c07-3260be21366b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Jul 20 10:11:25.291420 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:25.291380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc" event={"ID":"d67f084e-26e4-41d8-9620-1389b2cdf127","Type":"ContainerStarted","Data":"380b7542ad694dc043d68582a27fd611952eb646b85b2ec72bc47e74863b0412"}
Jul 20 10:11:25.313141 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:25.313073    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc" podStartSLOduration=1.6067651010000001 podStartE2EDuration="3.313052425s" totalImagesPullingTime="1.706287324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:11:25.309402985 +0000 UTC m=+897.358266635" watchObservedRunningTime="2026-07-20 10:11:25.313052425 +0000 UTC m=+897.361916073"
Jul 20 10:11:25.711895 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:25.711854    2580 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Jul 20 10:11:25.712188 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:25.712139    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp" podUID="06712755-7e3d-4b4c-9c07-3260be21366b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Jul 20 10:11:26.229665 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:26.229620    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:26.712098 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:26.712051    2580 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Jul 20 10:11:26.712329 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:26.712127    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp" podUID="06712755-7e3d-4b4c-9c07-3260be21366b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Jul 20 10:11:26.793157 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:26.793122    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:11:28.603150 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:28.603095    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:11:28.604898 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:28.604861    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-jfpwp"
Jul 20 10:11:33.229988 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:33.229951    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:33.242476 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:33.242450    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:33.343074 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:33.343046    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:11:42.726751 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:42.726712    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:53.636917 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:53.636874    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:56.424769 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:56.424730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:56.526385 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:11:56.526349    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:02.207604 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:02.207572    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"]
Jul 20 10:12:02.213857 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:02.213824    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:02.217265 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:12:02.217234    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:12:02.217265 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:12:02.217265 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:12:02.217265 ip-10-0-133-160 kubenswrapper[2580]:  > containerID="380b7542ad694dc043d68582a27fd611952eb646b85b2ec72bc47e74863b0412" cmd=["/bin/sleep","15"]
Jul 20 10:12:02.217584 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:02.217282    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc" message="" err=<
Jul 20 10:12:02.217584 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:12:02.217584 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:12:02.217584 ip-10-0-133-160 kubenswrapper[2580]:  >
Jul 20 10:12:02.217584 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:12:02.217294    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:12:02.217584 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:12:02.217584 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:12:02.217584 ip-10-0-133-160 kubenswrapper[2580]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc" podUID="d67f084e-26e4-41d8-9620-1389b2cdf127" containerName="main" containerID="cri-o://380b7542ad694dc043d68582a27fd611952eb646b85b2ec72bc47e74863b0412"
Jul 20 10:12:02.217584 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:02.217335    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc" podUID="d67f084e-26e4-41d8-9620-1389b2cdf127" containerName="main" containerID="cri-o://380b7542ad694dc043d68582a27fd611952eb646b85b2ec72bc47e74863b0412" gracePeriod=60
Jul 20 10:12:02.293974 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:02.293939    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:02.332066 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:02.332023    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc" podUID="d67f084e-26e4-41d8-9620-1389b2cdf127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 20 10:12:02.479573 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:02.479491    2580 generic.go:356] "Generic (PLEG): container finished" podID="d67f084e-26e4-41d8-9620-1389b2cdf127" containerID="380b7542ad694dc043d68582a27fd611952eb646b85b2ec72bc47e74863b0412" exitCode=0
Jul 20 10:12:02.479713 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:02.479563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc" event={"ID":"d67f084e-26e4-41d8-9620-1389b2cdf127","Type":"ContainerDied","Data":"380b7542ad694dc043d68582a27fd611952eb646b85b2ec72bc47e74863b0412"}
Jul 20 10:12:02.972792 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:02.972769    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:12:03.075108 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.075071    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8jjz\" (UniqueName: \"kubernetes.io/projected/d67f084e-26e4-41d8-9620-1389b2cdf127-kube-api-access-g8jjz\") pod \"d67f084e-26e4-41d8-9620-1389b2cdf127\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") "
Jul 20 10:12:03.075306 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.075114    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d67f084e-26e4-41d8-9620-1389b2cdf127-tls-certs\") pod \"d67f084e-26e4-41d8-9620-1389b2cdf127\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") "
Jul 20 10:12:03.075306 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.075208    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-home\") pod \"d67f084e-26e4-41d8-9620-1389b2cdf127\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") "
Jul 20 10:12:03.075306 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.075244    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-dshm\") pod \"d67f084e-26e4-41d8-9620-1389b2cdf127\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") "
Jul 20 10:12:03.075306 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.075270    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-tmp-dir\") pod \"d67f084e-26e4-41d8-9620-1389b2cdf127\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") "
Jul 20 10:12:03.075590 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.075309    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-model-cache\") pod \"d67f084e-26e4-41d8-9620-1389b2cdf127\" (UID: \"d67f084e-26e4-41d8-9620-1389b2cdf127\") "
Jul 20 10:12:03.075590 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.075531    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-home" (OuterVolumeSpecName: "home") pod "d67f084e-26e4-41d8-9620-1389b2cdf127" (UID: "d67f084e-26e4-41d8-9620-1389b2cdf127"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:12:03.075679 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.075612    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-home\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:12:03.075679 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.075611    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d67f084e-26e4-41d8-9620-1389b2cdf127" (UID: "d67f084e-26e4-41d8-9620-1389b2cdf127"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:12:03.075679 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.075671    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-model-cache" (OuterVolumeSpecName: "model-cache") pod "d67f084e-26e4-41d8-9620-1389b2cdf127" (UID: "d67f084e-26e4-41d8-9620-1389b2cdf127"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:12:03.077658 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.077632    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-dshm" (OuterVolumeSpecName: "dshm") pod "d67f084e-26e4-41d8-9620-1389b2cdf127" (UID: "d67f084e-26e4-41d8-9620-1389b2cdf127"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:12:03.077929 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.077909    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d67f084e-26e4-41d8-9620-1389b2cdf127-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d67f084e-26e4-41d8-9620-1389b2cdf127" (UID: "d67f084e-26e4-41d8-9620-1389b2cdf127"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:12:03.078036 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.077952    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67f084e-26e4-41d8-9620-1389b2cdf127-kube-api-access-g8jjz" (OuterVolumeSpecName: "kube-api-access-g8jjz") pod "d67f084e-26e4-41d8-9620-1389b2cdf127" (UID: "d67f084e-26e4-41d8-9620-1389b2cdf127"). InnerVolumeSpecName "kube-api-access-g8jjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:12:03.176537 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.176501    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8jjz\" (UniqueName: \"kubernetes.io/projected/d67f084e-26e4-41d8-9620-1389b2cdf127-kube-api-access-g8jjz\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:12:03.176537 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.176532    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d67f084e-26e4-41d8-9620-1389b2cdf127-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:12:03.176537 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.176551    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-dshm\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:12:03.176776 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.176561    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-tmp-dir\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:12:03.176776 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.176569    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d67f084e-26e4-41d8-9620-1389b2cdf127-model-cache\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:12:03.486507 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.486480    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"
Jul 20 10:12:03.486507 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.486483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc" event={"ID":"d67f084e-26e4-41d8-9620-1389b2cdf127","Type":"ContainerDied","Data":"abb3bb1686c34f1e407cb7592e1158be226918072cf34ee955c1b0a59bce7734"}
Jul 20 10:12:03.486997 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.486541    2580 scope.go:117] "RemoveContainer" containerID="380b7542ad694dc043d68582a27fd611952eb646b85b2ec72bc47e74863b0412"
Jul 20 10:12:03.513459 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.513427    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"]
Jul 20 10:12:03.515355 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:03.515334    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dfmzc"]
Jul 20 10:12:04.411051 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:04.411017    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d67f084e-26e4-41d8-9620-1389b2cdf127" path="/var/lib/kubelet/pods/d67f084e-26e4-41d8-9620-1389b2cdf127/volumes"
Jul 20 10:12:16.044652 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:16.044607    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:16.140071 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:16.140040    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:22.211443 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:22.211397    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:22.293449 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:22.293412    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:23.155788 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:23.155750    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:23.326575 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:23.326540    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:23.978070 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:23.978008    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:28.685810 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.685728    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"]
Jul 20 10:12:28.686428 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.686050    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d67f084e-26e4-41d8-9620-1389b2cdf127" containerName="main"
Jul 20 10:12:28.686428 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.686059    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d67f084e-26e4-41d8-9620-1389b2cdf127" containerName="main"
Jul 20 10:12:28.686428 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.686115    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d67f084e-26e4-41d8-9620-1389b2cdf127" containerName="main"
Jul 20 10:12:28.690055 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.690038    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.692948 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.692927    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Jul 20 10:12:28.693055 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.692931    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Jul 20 10:12:28.693055 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.692995    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Jul 20 10:12:28.693145 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.692931    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-lzhg5\""
Jul 20 10:12:28.699538 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.699516    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"]
Jul 20 10:12:28.786085 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.786051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-r26cn\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.786289 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.786099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6qzb\" (UniqueName: \"kubernetes.io/projected/2b80ae78-a06e-4193-905d-3f4e6c88e628-kube-api-access-t6qzb\") pod \"fc-concurrency-test-kserve-b5d44467c-r26cn\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.786289 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.786149    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-r26cn\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.786289 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.786191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b80ae78-a06e-4193-905d-3f4e6c88e628-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-r26cn\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.786289 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.786232    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-home\") pod \"fc-concurrency-test-kserve-b5d44467c-r26cn\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.786489 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.786315    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-r26cn\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.786489 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.786379    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:28.887127 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.887096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-r26cn\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.887127 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.887131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b80ae78-a06e-4193-905d-3f4e6c88e628-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-r26cn\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.887395 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.887305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-home\") pod \"fc-concurrency-test-kserve-b5d44467c-r26cn\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.887456 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.887408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-r26cn\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.887456 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.887445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-r26cn\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.887559 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.887487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6qzb\" (UniqueName: \"kubernetes.io/projected/2b80ae78-a06e-4193-905d-3f4e6c88e628-kube-api-access-t6qzb\") pod \"fc-concurrency-test-kserve-b5d44467c-r26cn\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.887559 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.887510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-r26cn\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.887658 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.887636    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-home\") pod \"fc-concurrency-test-kserve-b5d44467c-r26cn\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.887749 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.887728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-r26cn\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.889775 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.889745    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-r26cn\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.889885 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.889863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b80ae78-a06e-4193-905d-3f4e6c88e628-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-r26cn\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.898797 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.898769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6qzb\" (UniqueName: \"kubernetes.io/projected/2b80ae78-a06e-4193-905d-3f4e6c88e628-kube-api-access-t6qzb\") pod \"fc-concurrency-test-kserve-b5d44467c-r26cn\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:28.942353 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:28.942264    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:29.000583 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:29.000538    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:29.038072 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:29.036864    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:29.139045 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:29.139022    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"]
Jul 20 10:12:29.140913 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:12:29.140887    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b80ae78_a06e_4193_905d_3f4e6c88e628.slice/crio-007504f3cc9db1e73a61467f288d6c3484dd63dbda6f52ff9a63c136d1b6d729 WatchSource:0}: Error finding container 007504f3cc9db1e73a61467f288d6c3484dd63dbda6f52ff9a63c136d1b6d729: Status 404 returned error can't find the container with id 007504f3cc9db1e73a61467f288d6c3484dd63dbda6f52ff9a63c136d1b6d729
Jul 20 10:12:29.300893 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:29.300854    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:29.416582 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:29.416528    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:29.616206 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:29.616153    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn" event={"ID":"2b80ae78-a06e-4193-905d-3f4e6c88e628","Type":"ContainerStarted","Data":"e1fec0df59a658773dd13734ed83a6741915b9083aa1325e24d17ec6e7c0fcc4"}
Jul 20 10:12:29.616463 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:29.616247    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn" event={"ID":"2b80ae78-a06e-4193-905d-3f4e6c88e628","Type":"ContainerStarted","Data":"007504f3cc9db1e73a61467f288d6c3484dd63dbda6f52ff9a63c136d1b6d729"}
Jul 20 10:12:29.645261 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:29.645207    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn" podStartSLOduration=1.645191122 podStartE2EDuration="1.645191122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:12:29.641846868 +0000 UTC m=+961.690710525" watchObservedRunningTime="2026-07-20 10:12:29.645191122 +0000 UTC m=+961.694054769"
Jul 20 10:12:30.001685 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:30.001650    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:36.006039 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:36.005995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:36.111355 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:36.111324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:37.384628 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:37.384597    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:39.001506 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:39.001472    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:39.013688 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:39.013664    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:39.677321 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:39.677295    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:12:42.418660 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:42.418622    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:55.841810 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:12:55.841713    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:07.950162 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:07.950126    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"]
Jul 20 10:13:07.959179 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:13:07.959124    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:13:07.959179 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:13:07.959179 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:13:07.959179 ip-10-0-133-160 kubenswrapper[2580]:  > containerID="e1fec0df59a658773dd13734ed83a6741915b9083aa1325e24d17ec6e7c0fcc4" cmd=["/bin/sleep","15"]
Jul 20 10:13:07.959728 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:07.959197    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn" message="" err=<
Jul 20 10:13:07.959728 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:13:07.959728 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:13:07.959728 ip-10-0-133-160 kubenswrapper[2580]:  >
Jul 20 10:13:07.959728 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:13:07.959210    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:13:07.959728 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:13:07.959728 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:13:07.959728 ip-10-0-133-160 kubenswrapper[2580]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn" podUID="2b80ae78-a06e-4193-905d-3f4e6c88e628" containerName="main" containerID="cri-o://e1fec0df59a658773dd13734ed83a6741915b9083aa1325e24d17ec6e7c0fcc4"
Jul 20 10:13:07.959728 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:07.959249    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn" podUID="2b80ae78-a06e-4193-905d-3f4e6c88e628" containerName="main" containerID="cri-o://e1fec0df59a658773dd13734ed83a6741915b9083aa1325e24d17ec6e7c0fcc4" gracePeriod=60
Jul 20 10:13:07.960352 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:07.959718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:08.313448 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.313426    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:13:08.417798 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.417768    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-home\") pod \"2b80ae78-a06e-4193-905d-3f4e6c88e628\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") "
Jul 20 10:13:08.417967 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.417832    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-model-cache\") pod \"2b80ae78-a06e-4193-905d-3f4e6c88e628\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") "
Jul 20 10:13:08.417967 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.417887    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-dshm\") pod \"2b80ae78-a06e-4193-905d-3f4e6c88e628\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") "
Jul 20 10:13:08.417967 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.417922    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b80ae78-a06e-4193-905d-3f4e6c88e628-tls-certs\") pod \"2b80ae78-a06e-4193-905d-3f4e6c88e628\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") "
Jul 20 10:13:08.418114 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.417975    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6qzb\" (UniqueName: \"kubernetes.io/projected/2b80ae78-a06e-4193-905d-3f4e6c88e628-kube-api-access-t6qzb\") pod \"2b80ae78-a06e-4193-905d-3f4e6c88e628\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") "
Jul 20 10:13:08.418114 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.417998    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-tmp-dir\") pod \"2b80ae78-a06e-4193-905d-3f4e6c88e628\" (UID: \"2b80ae78-a06e-4193-905d-3f4e6c88e628\") "
Jul 20 10:13:08.418114 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.418048    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-home" (OuterVolumeSpecName: "home") pod "2b80ae78-a06e-4193-905d-3f4e6c88e628" (UID: "2b80ae78-a06e-4193-905d-3f4e6c88e628"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:13:08.418323 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.418116    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b80ae78-a06e-4193-905d-3f4e6c88e628" (UID: "2b80ae78-a06e-4193-905d-3f4e6c88e628"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:13:08.418323 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.418299    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-home\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:08.418423 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.418331    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-model-cache\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:08.418423 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.418297    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b80ae78-a06e-4193-905d-3f4e6c88e628" (UID: "2b80ae78-a06e-4193-905d-3f4e6c88e628"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:13:08.420438 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.420402    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b80ae78-a06e-4193-905d-3f4e6c88e628-kube-api-access-t6qzb" (OuterVolumeSpecName: "kube-api-access-t6qzb") pod "2b80ae78-a06e-4193-905d-3f4e6c88e628" (UID: "2b80ae78-a06e-4193-905d-3f4e6c88e628"). InnerVolumeSpecName "kube-api-access-t6qzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:13:08.420569 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.420466    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-dshm" (OuterVolumeSpecName: "dshm") pod "2b80ae78-a06e-4193-905d-3f4e6c88e628" (UID: "2b80ae78-a06e-4193-905d-3f4e6c88e628"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:13:08.420569 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.420526    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b80ae78-a06e-4193-905d-3f4e6c88e628-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b80ae78-a06e-4193-905d-3f4e6c88e628" (UID: "2b80ae78-a06e-4193-905d-3f4e6c88e628"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:13:08.519338 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.519233    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-dshm\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:08.519539 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.519357    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b80ae78-a06e-4193-905d-3f4e6c88e628-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:08.519539 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.519377    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6qzb\" (UniqueName: \"kubernetes.io/projected/2b80ae78-a06e-4193-905d-3f4e6c88e628-kube-api-access-t6qzb\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:08.519539 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.519390    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b80ae78-a06e-4193-905d-3f4e6c88e628-tmp-dir\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:08.814976 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.814884    2580 generic.go:356] "Generic (PLEG): container finished" podID="2b80ae78-a06e-4193-905d-3f4e6c88e628" containerID="e1fec0df59a658773dd13734ed83a6741915b9083aa1325e24d17ec6e7c0fcc4" exitCode=0
Jul 20 10:13:08.814976 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.814931    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"
Jul 20 10:13:08.815157 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.814977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn" event={"ID":"2b80ae78-a06e-4193-905d-3f4e6c88e628","Type":"ContainerDied","Data":"e1fec0df59a658773dd13734ed83a6741915b9083aa1325e24d17ec6e7c0fcc4"}
Jul 20 10:13:08.815157 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.815024    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn" event={"ID":"2b80ae78-a06e-4193-905d-3f4e6c88e628","Type":"ContainerDied","Data":"007504f3cc9db1e73a61467f288d6c3484dd63dbda6f52ff9a63c136d1b6d729"}
Jul 20 10:13:08.815157 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.815048    2580 scope.go:117] "RemoveContainer" containerID="e1fec0df59a658773dd13734ed83a6741915b9083aa1325e24d17ec6e7c0fcc4"
Jul 20 10:13:08.827082 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.827057    2580 scope.go:117] "RemoveContainer" containerID="e1fec0df59a658773dd13734ed83a6741915b9083aa1325e24d17ec6e7c0fcc4"
Jul 20 10:13:08.827370 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:13:08.827351    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1fec0df59a658773dd13734ed83a6741915b9083aa1325e24d17ec6e7c0fcc4\": container with ID starting with e1fec0df59a658773dd13734ed83a6741915b9083aa1325e24d17ec6e7c0fcc4 not found: ID does not exist" containerID="e1fec0df59a658773dd13734ed83a6741915b9083aa1325e24d17ec6e7c0fcc4"
Jul 20 10:13:08.827429 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.827379    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1fec0df59a658773dd13734ed83a6741915b9083aa1325e24d17ec6e7c0fcc4"} err="failed to get container status \"e1fec0df59a658773dd13734ed83a6741915b9083aa1325e24d17ec6e7c0fcc4\": rpc error: code = NotFound desc = could not find container \"e1fec0df59a658773dd13734ed83a6741915b9083aa1325e24d17ec6e7c0fcc4\": container with ID starting with e1fec0df59a658773dd13734ed83a6741915b9083aa1325e24d17ec6e7c0fcc4 not found: ID does not exist"
Jul 20 10:13:08.846697 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.846639    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"]
Jul 20 10:13:08.848731 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:08.848706    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-r26cn"]
Jul 20 10:13:09.547709 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:09.547679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:10.410777 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:10.410744    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b80ae78-a06e-4193-905d-3f4e6c88e628" path="/var/lib/kubelet/pods/2b80ae78-a06e-4193-905d-3f4e6c88e628/volumes"
Jul 20 10:13:27.877943 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:27.877904    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:27.986154 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:27.986119    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:28.090669 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.090635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:28.183662 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.183581    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"]
Jul 20 10:13:28.184305 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.184282    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b80ae78-a06e-4193-905d-3f4e6c88e628" containerName="main"
Jul 20 10:13:28.184305 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.184306    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b80ae78-a06e-4193-905d-3f4e6c88e628" containerName="main"
Jul 20 10:13:28.184473 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.184394    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b80ae78-a06e-4193-905d-3f4e6c88e628" containerName="main"
Jul 20 10:13:28.187763 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.187737    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.192497 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.192472    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Jul 20 10:13:28.193088 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.193063    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Jul 20 10:13:28.193299 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.193276    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-zjdrq\""
Jul 20 10:13:28.193421 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.193320    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Jul 20 10:13:28.213438 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.213401    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"]
Jul 20 10:13:28.266434 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.266390    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:28.290948 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.290921    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-data\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.291100 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.290955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-podinfo\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.291100 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.290979    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.291100 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.290997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-workload-socket\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.291100 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.291060    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-credential-socket\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.291346 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.291099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-envoy\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.291346 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.291129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-workload-certs\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.291346 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.291183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-token\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.291346 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.291231    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fklz\" (UniqueName: \"kubernetes.io/projected/c578a67f-4584-4cf9-b5d7-533f3e96ac51-kube-api-access-9fklz\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.368878 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.368834    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:28.392682 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.392619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-workload-socket\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.392682 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.392673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-credential-socket\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.392937 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.392715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-envoy\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.392937 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.392749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-workload-certs\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.392937 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.392777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-token\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.392937 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.392811    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fklz\" (UniqueName: \"kubernetes.io/projected/c578a67f-4584-4cf9-b5d7-533f3e96ac51-kube-api-access-9fklz\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.392937 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.392844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-data\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.392937 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.392871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-podinfo\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.393275 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.392956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.393275 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.393047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-workload-socket\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.393275 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.393085    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-credential-socket\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.393275 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.393191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-workload-certs\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.393478 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.393396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-data\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.395588 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.395562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-envoy\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.395738 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.395717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-podinfo\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.396988 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.396965    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Jul 20 10:13:28.404318 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.403885    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.405802 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.405536    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Jul 20 10:13:28.405915 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.405899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-token\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.416653 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.416631    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Jul 20 10:13:28.425788 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.425766    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fklz\" (UniqueName: \"kubernetes.io/projected/c578a67f-4584-4cf9-b5d7-533f3e96ac51-kube-api-access-9fklz\") pod \"router-gateway-1-openshift-default-686f699649-8hpxw\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.502797 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.502769    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-zjdrq\""
Jul 20 10:13:28.511415 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.511398    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:28.671351 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.671328    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"]
Jul 20 10:13:28.673735 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:13:28.673710    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc578a67f_4584_4cf9_b5d7_533f3e96ac51.slice/crio-1a43dd5ea420c1b21437dbcc20ca1593718c397f55eab7644d58028cdc1e11ca WatchSource:0}: Error finding container 1a43dd5ea420c1b21437dbcc20ca1593718c397f55eab7644d58028cdc1e11ca: Status 404 returned error can't find the container with id 1a43dd5ea420c1b21437dbcc20ca1593718c397f55eab7644d58028cdc1e11ca
Jul 20 10:13:28.675694 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.675662    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 20 10:13:28.675748 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.675727    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 20 10:13:28.675789 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.675776    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 20 10:13:28.922311 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.922207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw" event={"ID":"c578a67f-4584-4cf9-b5d7-533f3e96ac51","Type":"ContainerStarted","Data":"5e08f2ce3e734ef32088fa11ba54a8519aea009f91cea88e43458bc8970b2760"}
Jul 20 10:13:28.922311 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.922270    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw" event={"ID":"c578a67f-4584-4cf9-b5d7-533f3e96ac51","Type":"ContainerStarted","Data":"1a43dd5ea420c1b21437dbcc20ca1593718c397f55eab7644d58028cdc1e11ca"}
Jul 20 10:13:28.974552 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:28.974498    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw" podStartSLOduration=0.974482684 podStartE2EDuration="974.482684ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:13:28.972746767 +0000 UTC m=+1021.021610415" watchObservedRunningTime="2026-07-20 10:13:28.974482684 +0000 UTC m=+1021.023346331"
Jul 20 10:13:29.512101 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:29.512054    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:29.517296 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:29.517268    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:29.927641 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:29.927547    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:29.928406 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:29.928389    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:31.898918 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:31.898884    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:34.750040 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:34.750001    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:35.091393 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.091315    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"]
Jul 20 10:13:35.094850 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.094831    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.098321 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.098301    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 20 10:13:35.098449 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.098302    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-mxkcl\""
Jul 20 10:13:35.108424 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.108400    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"]
Jul 20 10:13:35.134746 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.134719    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:35.153813 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.153785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhh57\" (UniqueName: \"kubernetes.io/projected/38a4ddd9-6c77-4c15-957b-a59b32698567-kube-api-access-nhh57\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.153934 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.153820    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.153934 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.153843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a4ddd9-6c77-4c15-957b-a59b32698567-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.153934 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.153859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.154042 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.153929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.154042 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.154000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.255292 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.255253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhh57\" (UniqueName: \"kubernetes.io/projected/38a4ddd9-6c77-4c15-957b-a59b32698567-kube-api-access-nhh57\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.255479 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.255305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.255479 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.255341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a4ddd9-6c77-4c15-957b-a59b32698567-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.255479 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.255367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.255479 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.255391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.255479 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.255425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.255741 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.255730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.255829 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.255809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.255885 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.255834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.257771 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.257754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.258086 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.258068    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a4ddd9-6c77-4c15-957b-a59b32698567-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.264849 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.264827    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhh57\" (UniqueName: \"kubernetes.io/projected/38a4ddd9-6c77-4c15-957b-a59b32698567-kube-api-access-nhh57\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.406792 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.406700    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:35.548190 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.548144    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"]
Jul 20 10:13:35.550423 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:13:35.550397    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a4ddd9_6c77_4c15_957b_a59b32698567.slice/crio-6b70ec6dbe6d48be645baf097a46392e69fdd9cba2a8945b0f1d82dc9cc293f7 WatchSource:0}: Error finding container 6b70ec6dbe6d48be645baf097a46392e69fdd9cba2a8945b0f1d82dc9cc293f7: Status 404 returned error can't find the container with id 6b70ec6dbe6d48be645baf097a46392e69fdd9cba2a8945b0f1d82dc9cc293f7
Jul 20 10:13:35.959557 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.959522    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr" event={"ID":"38a4ddd9-6c77-4c15-957b-a59b32698567","Type":"ContainerStarted","Data":"2a4f5d9fc4e866c7b0ceec0620c50eacfbdc90cb011f6179807ae29d21d5e57e"}
Jul 20 10:13:35.959946 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.959603    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr" event={"ID":"38a4ddd9-6c77-4c15-957b-a59b32698567","Type":"ContainerStarted","Data":"6b70ec6dbe6d48be645baf097a46392e69fdd9cba2a8945b0f1d82dc9cc293f7"}
Jul 20 10:13:35.983327 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:35.983282    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr" podStartSLOduration=0.983265014 podStartE2EDuration="983.265014ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:13:35.98215748 +0000 UTC m=+1028.031021129" watchObservedRunningTime="2026-07-20 10:13:35.983265014 +0000 UTC m=+1028.032128660"
Jul 20 10:13:36.414738 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:36.414612    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:37.327241 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.327198    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:37.522503 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.522461    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:37.640720 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.640619    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:37.690839 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.690798    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"]
Jul 20 10:13:37.694696 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.694672    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.697655 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.697634    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Jul 20 10:13:37.697772 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.697736    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-wg8sj\""
Jul 20 10:13:37.698465 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.698446    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Jul 20 10:13:37.721191 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.715434    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Jul 20 10:13:37.730193 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.726281    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"]
Jul 20 10:13:37.743807 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.743784    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:37.781154 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.781123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e0b47e-81f1-4139-941b-76d44d85927a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.781306 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.781284    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-credential-socket\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.781362 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.781312    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-workload-certs\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.781362 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.781332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2sxq\" (UniqueName: \"kubernetes.io/projected/19e0b47e-81f1-4139-941b-76d44d85927a-kube-api-access-z2sxq\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.781362 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.781349    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-istio-envoy\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.781489 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.781366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-istio-data\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.781532 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.781483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e0b47e-81f1-4139-941b-76d44d85927a-istio-podinfo\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.781532 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.781517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e0b47e-81f1-4139-941b-76d44d85927a-istio-token\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.781607 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.781537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-workload-socket\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.882451 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.882417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-credential-socket\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.882451 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.882454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-workload-certs\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.882779 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.882472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2sxq\" (UniqueName: \"kubernetes.io/projected/19e0b47e-81f1-4139-941b-76d44d85927a-kube-api-access-z2sxq\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.882779 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.882492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-istio-envoy\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.882779 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.882510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-istio-data\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.882779 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.882544    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e0b47e-81f1-4139-941b-76d44d85927a-istio-podinfo\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.882779 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.882580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e0b47e-81f1-4139-941b-76d44d85927a-istio-token\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.882779 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.882609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-workload-socket\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.882779 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.882638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e0b47e-81f1-4139-941b-76d44d85927a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.883135 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.882931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-workload-certs\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.883135 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.882959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-istio-data\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.883264 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.883140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-credential-socket\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.883421 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.883404    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e0b47e-81f1-4139-941b-76d44d85927a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.883469 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.883448    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-workload-socket\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.884957 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.884939    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-istio-envoy\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.885158 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.885143    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e0b47e-81f1-4139-941b-76d44d85927a-istio-podinfo\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.893818 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.893755    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e0b47e-81f1-4139-941b-76d44d85927a-istio-token\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.894251 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.894230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2sxq\" (UniqueName: \"kubernetes.io/projected/19e0b47e-81f1-4139-941b-76d44d85927a-kube-api-access-z2sxq\") pod \"router-gateway-1-openshift-default-84cf497d49-qwjsg\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:37.926122 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:37.926080    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:38.034397 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:38.034350    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:38.222673 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:38.222630    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"]
Jul 20 10:13:38.226543 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:13:38.226510    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e0b47e_81f1_4139_941b_76d44d85927a.slice/crio-af50de795c095de406a934bf7928cdd949bd1223982c84908b437d9af3aba019 WatchSource:0}: Error finding container af50de795c095de406a934bf7928cdd949bd1223982c84908b437d9af3aba019: Status 404 returned error can't find the container with id af50de795c095de406a934bf7928cdd949bd1223982c84908b437d9af3aba019
Jul 20 10:13:38.228642 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:38.228608    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 20 10:13:38.228742 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:38.228698    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 20 10:13:38.228742 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:38.228727    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 20 10:13:38.975793 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:38.975751    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg" event={"ID":"19e0b47e-81f1-4139-941b-76d44d85927a","Type":"ContainerStarted","Data":"16fe77c702fdba917cd9e86c8c72fa82235ebed3b4e61d4a9e1c9fd904ca0806"}
Jul 20 10:13:38.976301 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:38.975808    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg" event={"ID":"19e0b47e-81f1-4139-941b-76d44d85927a","Type":"ContainerStarted","Data":"af50de795c095de406a934bf7928cdd949bd1223982c84908b437d9af3aba019"}
Jul 20 10:13:39.000606 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:39.000536    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg" podStartSLOduration=2.000517031 podStartE2EDuration="2.000517031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:13:38.998948468 +0000 UTC m=+1031.047812116" watchObservedRunningTime="2026-07-20 10:13:39.000517031 +0000 UTC m=+1031.049380681"
Jul 20 10:13:39.035472 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:39.035223    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:39.046621 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:39.046594    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:39.981483 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:39.981455    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:39.982527 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:39.982504    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:13:40.313250 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.308850    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"]
Jul 20 10:13:40.330749 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.325743    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:40.335660 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:13:40.335606    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:13:40.335660 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:13:40.335660 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:13:40.335660 ip-10-0-133-160 kubenswrapper[2580]:  > containerID="2a4f5d9fc4e866c7b0ceec0620c50eacfbdc90cb011f6179807ae29d21d5e57e" cmd=["/bin/sleep","15"]
Jul 20 10:13:40.335871 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.335697    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr" message="" err=<
Jul 20 10:13:40.335871 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:13:40.335871 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:13:40.335871 ip-10-0-133-160 kubenswrapper[2580]:  >
Jul 20 10:13:40.335871 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:13:40.335727    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:13:40.335871 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:13:40.335871 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:13:40.335871 ip-10-0-133-160 kubenswrapper[2580]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr" podUID="38a4ddd9-6c77-4c15-957b-a59b32698567" containerName="main" containerID="cri-o://2a4f5d9fc4e866c7b0ceec0620c50eacfbdc90cb011f6179807ae29d21d5e57e"
Jul 20 10:13:40.335871 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.335793    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr" podUID="38a4ddd9-6c77-4c15-957b-a59b32698567" containerName="main" containerID="cri-o://2a4f5d9fc4e866c7b0ceec0620c50eacfbdc90cb011f6179807ae29d21d5e57e" gracePeriod=60
Jul 20 10:13:40.631711 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.631573    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:40.724446 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.724411    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhh57\" (UniqueName: \"kubernetes.io/projected/38a4ddd9-6c77-4c15-957b-a59b32698567-kube-api-access-nhh57\") pod \"38a4ddd9-6c77-4c15-957b-a59b32698567\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") "
Jul 20 10:13:40.724626 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.724493    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-dshm\") pod \"38a4ddd9-6c77-4c15-957b-a59b32698567\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") "
Jul 20 10:13:40.724626 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.724542    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-tmp-dir\") pod \"38a4ddd9-6c77-4c15-957b-a59b32698567\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") "
Jul 20 10:13:40.724626 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.724578    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-model-cache\") pod \"38a4ddd9-6c77-4c15-957b-a59b32698567\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") "
Jul 20 10:13:40.724626 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.724604    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a4ddd9-6c77-4c15-957b-a59b32698567-tls-certs\") pod \"38a4ddd9-6c77-4c15-957b-a59b32698567\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") "
Jul 20 10:13:40.724770 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.724651    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-home\") pod \"38a4ddd9-6c77-4c15-957b-a59b32698567\" (UID: \"38a4ddd9-6c77-4c15-957b-a59b32698567\") "
Jul 20 10:13:40.724801 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.724769    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38a4ddd9-6c77-4c15-957b-a59b32698567" (UID: "38a4ddd9-6c77-4c15-957b-a59b32698567"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:13:40.724904 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.724867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-model-cache" (OuterVolumeSpecName: "model-cache") pod "38a4ddd9-6c77-4c15-957b-a59b32698567" (UID: "38a4ddd9-6c77-4c15-957b-a59b32698567"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:13:40.725068 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.724936    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-tmp-dir\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:40.725068 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.724956    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-home" (OuterVolumeSpecName: "home") pod "38a4ddd9-6c77-4c15-957b-a59b32698567" (UID: "38a4ddd9-6c77-4c15-957b-a59b32698567"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:13:40.727070 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.727049    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38a4ddd9-6c77-4c15-957b-a59b32698567-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38a4ddd9-6c77-4c15-957b-a59b32698567" (UID: "38a4ddd9-6c77-4c15-957b-a59b32698567"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:13:40.727246 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.727218    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-dshm" (OuterVolumeSpecName: "dshm") pod "38a4ddd9-6c77-4c15-957b-a59b32698567" (UID: "38a4ddd9-6c77-4c15-957b-a59b32698567"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:13:40.727325 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.727267    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a4ddd9-6c77-4c15-957b-a59b32698567-kube-api-access-nhh57" (OuterVolumeSpecName: "kube-api-access-nhh57") pod "38a4ddd9-6c77-4c15-957b-a59b32698567" (UID: "38a4ddd9-6c77-4c15-957b-a59b32698567"). InnerVolumeSpecName "kube-api-access-nhh57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:13:40.826181 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.826121    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-model-cache\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:40.826181 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.826177    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a4ddd9-6c77-4c15-957b-a59b32698567-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:40.826181 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.826192    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-home\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:40.826438 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.826206    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhh57\" (UniqueName: \"kubernetes.io/projected/38a4ddd9-6c77-4c15-957b-a59b32698567-kube-api-access-nhh57\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:40.826438 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.826219    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38a4ddd9-6c77-4c15-957b-a59b32698567-dshm\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:40.987523 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.987489    2580 generic.go:356] "Generic (PLEG): container finished" podID="38a4ddd9-6c77-4c15-957b-a59b32698567" containerID="2a4f5d9fc4e866c7b0ceec0620c50eacfbdc90cb011f6179807ae29d21d5e57e" exitCode=0
Jul 20 10:13:40.987956 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.987551    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"
Jul 20 10:13:40.987956 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.987549    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr" event={"ID":"38a4ddd9-6c77-4c15-957b-a59b32698567","Type":"ContainerDied","Data":"2a4f5d9fc4e866c7b0ceec0620c50eacfbdc90cb011f6179807ae29d21d5e57e"}
Jul 20 10:13:40.987956 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.987606    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr" event={"ID":"38a4ddd9-6c77-4c15-957b-a59b32698567","Type":"ContainerDied","Data":"6b70ec6dbe6d48be645baf097a46392e69fdd9cba2a8945b0f1d82dc9cc293f7"}
Jul 20 10:13:40.987956 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.987630    2580 scope.go:117] "RemoveContainer" containerID="2a4f5d9fc4e866c7b0ceec0620c50eacfbdc90cb011f6179807ae29d21d5e57e"
Jul 20 10:13:40.998063 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.998041    2580 scope.go:117] "RemoveContainer" containerID="2a4f5d9fc4e866c7b0ceec0620c50eacfbdc90cb011f6179807ae29d21d5e57e"
Jul 20 10:13:40.998373 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:13:40.998351    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4f5d9fc4e866c7b0ceec0620c50eacfbdc90cb011f6179807ae29d21d5e57e\": container with ID starting with 2a4f5d9fc4e866c7b0ceec0620c50eacfbdc90cb011f6179807ae29d21d5e57e not found: ID does not exist" containerID="2a4f5d9fc4e866c7b0ceec0620c50eacfbdc90cb011f6179807ae29d21d5e57e"
Jul 20 10:13:40.998443 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:40.998385    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4f5d9fc4e866c7b0ceec0620c50eacfbdc90cb011f6179807ae29d21d5e57e"} err="failed to get container status \"2a4f5d9fc4e866c7b0ceec0620c50eacfbdc90cb011f6179807ae29d21d5e57e\": rpc error: code = NotFound desc = could not find container \"2a4f5d9fc4e866c7b0ceec0620c50eacfbdc90cb011f6179807ae29d21d5e57e\": container with ID starting with 2a4f5d9fc4e866c7b0ceec0620c50eacfbdc90cb011f6179807ae29d21d5e57e not found: ID does not exist"
Jul 20 10:13:41.012750 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:41.012714    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"]
Jul 20 10:13:41.019105 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:41.019073    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6xjhr"]
Jul 20 10:13:41.330465 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:41.330432    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:41.625774 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:41.625738    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:41.722055 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:41.722022    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:42.334554 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:42.334518    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"]
Jul 20 10:13:42.334992 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:42.334775    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw" podUID="c578a67f-4584-4cf9-b5d7-533f3e96ac51" containerName="istio-proxy" containerID="cri-o://5e08f2ce3e734ef32088fa11ba54a8519aea009f91cea88e43458bc8970b2760" gracePeriod=30
Jul 20 10:13:42.413209 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:42.413146    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38a4ddd9-6c77-4c15-957b-a59b32698567" path="/var/lib/kubelet/pods/38a4ddd9-6c77-4c15-957b-a59b32698567/volumes"
Jul 20 10:13:42.440364 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:42.440321    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:44.689057 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.689033    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:44.758968 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.758934    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-token\") pod \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") "
Jul 20 10:13:44.759120 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.758998    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-podinfo\") pod \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") "
Jul 20 10:13:44.759226 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.759186    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istiod-ca-cert\") pod \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") "
Jul 20 10:13:44.759294 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.759243    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-envoy\") pod \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") "
Jul 20 10:13:44.759294 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.759290    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-data\") pod \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") "
Jul 20 10:13:44.759405 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.759316    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-workload-certs\") pod \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") "
Jul 20 10:13:44.759405 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.759373    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-credential-socket\") pod \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") "
Jul 20 10:13:44.759571 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.759414    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-workload-socket\") pod \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") "
Jul 20 10:13:44.759571 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.759479    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fklz\" (UniqueName: \"kubernetes.io/projected/c578a67f-4584-4cf9-b5d7-533f3e96ac51-kube-api-access-9fklz\") pod \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\" (UID: \"c578a67f-4584-4cf9-b5d7-533f3e96ac51\") "
Jul 20 10:13:44.759935 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.759594    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c578a67f-4584-4cf9-b5d7-533f3e96ac51" (UID: "c578a67f-4584-4cf9-b5d7-533f3e96ac51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:13:44.759935 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.759768    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-data" (OuterVolumeSpecName: "istio-data") pod "c578a67f-4584-4cf9-b5d7-533f3e96ac51" (UID: "c578a67f-4584-4cf9-b5d7-533f3e96ac51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:13:44.759935 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.759774    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c578a67f-4584-4cf9-b5d7-533f3e96ac51" (UID: "c578a67f-4584-4cf9-b5d7-533f3e96ac51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:13:44.759935 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.759714    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c578a67f-4584-4cf9-b5d7-533f3e96ac51" (UID: "c578a67f-4584-4cf9-b5d7-533f3e96ac51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:13:44.760215 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.759987    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c578a67f-4584-4cf9-b5d7-533f3e96ac51" (UID: "c578a67f-4584-4cf9-b5d7-533f3e96ac51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:13:44.760215 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.760026    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-data\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:44.760215 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.760046    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-workload-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:44.760215 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.760061    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-credential-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:44.760215 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.760097    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istiod-ca-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:44.761781 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.761751    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c578a67f-4584-4cf9-b5d7-533f3e96ac51" (UID: "c578a67f-4584-4cf9-b5d7-533f3e96ac51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 20 10:13:44.762041 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.761933    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-token" (OuterVolumeSpecName: "istio-token") pod "c578a67f-4584-4cf9-b5d7-533f3e96ac51" (UID: "c578a67f-4584-4cf9-b5d7-533f3e96ac51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:13:44.762106 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.762086    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c578a67f-4584-4cf9-b5d7-533f3e96ac51" (UID: "c578a67f-4584-4cf9-b5d7-533f3e96ac51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:13:44.762263 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.762241    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c578a67f-4584-4cf9-b5d7-533f3e96ac51-kube-api-access-9fklz" (OuterVolumeSpecName: "kube-api-access-9fklz") pod "c578a67f-4584-4cf9-b5d7-533f3e96ac51" (UID: "c578a67f-4584-4cf9-b5d7-533f3e96ac51"). InnerVolumeSpecName "kube-api-access-9fklz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:13:44.861017 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.860974    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-workload-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:44.861017 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.861011    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fklz\" (UniqueName: \"kubernetes.io/projected/c578a67f-4584-4cf9-b5d7-533f3e96ac51-kube-api-access-9fklz\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:44.861017 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.861022    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-token\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:44.861297 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.861030    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-podinfo\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:44.861297 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:44.861038    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c578a67f-4584-4cf9-b5d7-533f3e96ac51-istio-envoy\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:13:45.017040 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:45.016949    2580 generic.go:356] "Generic (PLEG): container finished" podID="c578a67f-4584-4cf9-b5d7-533f3e96ac51" containerID="5e08f2ce3e734ef32088fa11ba54a8519aea009f91cea88e43458bc8970b2760" exitCode=0
Jul 20 10:13:45.017040 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:45.016991    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"
Jul 20 10:13:45.017040 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:45.016997    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw" event={"ID":"c578a67f-4584-4cf9-b5d7-533f3e96ac51","Type":"ContainerDied","Data":"5e08f2ce3e734ef32088fa11ba54a8519aea009f91cea88e43458bc8970b2760"}
Jul 20 10:13:45.017321 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:45.017048    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw" event={"ID":"c578a67f-4584-4cf9-b5d7-533f3e96ac51","Type":"ContainerDied","Data":"1a43dd5ea420c1b21437dbcc20ca1593718c397f55eab7644d58028cdc1e11ca"}
Jul 20 10:13:45.017321 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:45.017070    2580 scope.go:117] "RemoveContainer" containerID="5e08f2ce3e734ef32088fa11ba54a8519aea009f91cea88e43458bc8970b2760"
Jul 20 10:13:45.026470 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:45.026452    2580 scope.go:117] "RemoveContainer" containerID="5e08f2ce3e734ef32088fa11ba54a8519aea009f91cea88e43458bc8970b2760"
Jul 20 10:13:45.026727 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:13:45.026709    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e08f2ce3e734ef32088fa11ba54a8519aea009f91cea88e43458bc8970b2760\": container with ID starting with 5e08f2ce3e734ef32088fa11ba54a8519aea009f91cea88e43458bc8970b2760 not found: ID does not exist" containerID="5e08f2ce3e734ef32088fa11ba54a8519aea009f91cea88e43458bc8970b2760"
Jul 20 10:13:45.026772 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:45.026736    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e08f2ce3e734ef32088fa11ba54a8519aea009f91cea88e43458bc8970b2760"} err="failed to get container status \"5e08f2ce3e734ef32088fa11ba54a8519aea009f91cea88e43458bc8970b2760\": rpc error: code = NotFound desc = could not find container \"5e08f2ce3e734ef32088fa11ba54a8519aea009f91cea88e43458bc8970b2760\": container with ID starting with 5e08f2ce3e734ef32088fa11ba54a8519aea009f91cea88e43458bc8970b2760 not found: ID does not exist"
Jul 20 10:13:45.038016 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:45.037992    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"]
Jul 20 10:13:45.041547 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:45.041529    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-686f699649-8hpxw"]
Jul 20 10:13:46.411484 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:46.411453    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c578a67f-4584-4cf9-b5d7-533f3e96ac51" path="/var/lib/kubelet/pods/c578a67f-4584-4cf9-b5d7-533f3e96ac51/volumes"
Jul 20 10:13:47.566384 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:47.566352    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:48.674630 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:48.674533    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:54.003670 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:54.003622    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:54.478845 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:54.478812    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:54.580781 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:54.580745    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:55.896158 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:55.896121    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"]
Jul 20 10:13:55.896567 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:55.896471    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a4ddd9-6c77-4c15-957b-a59b32698567" containerName="main"
Jul 20 10:13:55.896567 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:55.896483    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a4ddd9-6c77-4c15-957b-a59b32698567" containerName="main"
Jul 20 10:13:55.896567 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:55.896502    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c578a67f-4584-4cf9-b5d7-533f3e96ac51" containerName="istio-proxy"
Jul 20 10:13:55.896567 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:55.896508    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c578a67f-4584-4cf9-b5d7-533f3e96ac51" containerName="istio-proxy"
Jul 20 10:13:55.896706 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:55.896602    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="38a4ddd9-6c77-4c15-957b-a59b32698567" containerName="main"
Jul 20 10:13:55.896706 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:55.896617    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c578a67f-4584-4cf9-b5d7-533f3e96ac51" containerName="istio-proxy"
Jul 20 10:13:55.898755 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:55.898735    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:55.901431 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:55.901406    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-qv6qd\""
Jul 20 10:13:55.901564 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:55.901406    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Jul 20 10:13:55.911057 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:55.911034    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"]
Jul 20 10:13:56.016906 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.016861    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:56.063897 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.063862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.063897 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.063909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.064143 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.063959    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.064143 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.063999    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjrdk\" (UniqueName: \"kubernetes.io/projected/8a0e310f-9315-497a-bbd1-e6ce15a90119-kube-api-access-fjrdk\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.064143 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.064025    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a0e310f-9315-497a-bbd1-e6ce15a90119-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.064143 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.064057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.164611 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.164523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjrdk\" (UniqueName: \"kubernetes.io/projected/8a0e310f-9315-497a-bbd1-e6ce15a90119-kube-api-access-fjrdk\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.164611 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.164575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a0e310f-9315-497a-bbd1-e6ce15a90119-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.164611 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.164611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.164885 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.164639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.164885 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.164681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.164885 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.164719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.165053 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.165033    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.165114 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.165091    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.165195 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.165131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.167220 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.167199    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.167500 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.167482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a0e310f-9315-497a-bbd1-e6ce15a90119-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.172982 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.172959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjrdk\" (UniqueName: \"kubernetes.io/projected/8a0e310f-9315-497a-bbd1-e6ce15a90119-kube-api-access-fjrdk\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.209032 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.208993    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:56.353698 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:56.353664    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"]
Jul 20 10:13:56.357734 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:13:56.357711    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0e310f_9315_497a_bbd1_e6ce15a90119.slice/crio-5651a1574f17745d8cffc03112c12ccf65f84621c2aa11d9e13d5d05d74cdf8d WatchSource:0}: Error finding container 5651a1574f17745d8cffc03112c12ccf65f84621c2aa11d9e13d5d05d74cdf8d: Status 404 returned error can't find the container with id 5651a1574f17745d8cffc03112c12ccf65f84621c2aa11d9e13d5d05d74cdf8d
Jul 20 10:13:57.082467 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:57.082433    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw" event={"ID":"8a0e310f-9315-497a-bbd1-e6ce15a90119","Type":"ContainerStarted","Data":"9f805eab6bad541f96eaac2e0cfa4dc53189ddf3107855d173f3ebba337bd42b"}
Jul 20 10:13:57.082879 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:57.082510    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw" event={"ID":"8a0e310f-9315-497a-bbd1-e6ce15a90119","Type":"ContainerStarted","Data":"5651a1574f17745d8cffc03112c12ccf65f84621c2aa11d9e13d5d05d74cdf8d"}
Jul 20 10:13:57.100850 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:57.100785    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw" podStartSLOduration=2.100765383 podStartE2EDuration="2.100765383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:13:57.097634055 +0000 UTC m=+1049.146497701" watchObservedRunningTime="2026-07-20 10:13:57.100765383 +0000 UTC m=+1049.149629031"
Jul 20 10:13:57.209202 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:57.209145    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:13:58.356484 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:58.356452    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:58.453925 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:58.453884    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:13:59.295747 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:59.295709    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"]
Jul 20 10:13:59.320710 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:13:59.320673    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:00.126427 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:14:00.126386    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:14:00.126427 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:14:00.126427 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:14:00.126427 ip-10-0-133-160 kubenswrapper[2580]:  > containerID="9f805eab6bad541f96eaac2e0cfa4dc53189ddf3107855d173f3ebba337bd42b" cmd=["/bin/sleep","15"]
Jul 20 10:14:00.127074 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.126443    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw" message="" err=<
Jul 20 10:14:00.127074 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:14:00.127074 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:14:00.127074 ip-10-0-133-160 kubenswrapper[2580]:  >
Jul 20 10:14:00.127074 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:14:00.126457    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:14:00.127074 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:14:00.127074 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:14:00.127074 ip-10-0-133-160 kubenswrapper[2580]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw" podUID="8a0e310f-9315-497a-bbd1-e6ce15a90119" containerName="main" containerID="cri-o://9f805eab6bad541f96eaac2e0cfa4dc53189ddf3107855d173f3ebba337bd42b"
Jul 20 10:14:00.127074 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.126501    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw" podUID="8a0e310f-9315-497a-bbd1-e6ce15a90119" containerName="main" containerID="cri-o://9f805eab6bad541f96eaac2e0cfa4dc53189ddf3107855d173f3ebba337bd42b" gracePeriod=60
Jul 20 10:14:00.380268 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.380207    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:14:00.420192 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.418747    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-home\") pod \"8a0e310f-9315-497a-bbd1-e6ce15a90119\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") "
Jul 20 10:14:00.420192 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.418818    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-model-cache\") pod \"8a0e310f-9315-497a-bbd1-e6ce15a90119\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") "
Jul 20 10:14:00.420192 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.418866    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-dshm\") pod \"8a0e310f-9315-497a-bbd1-e6ce15a90119\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") "
Jul 20 10:14:00.420192 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.418893    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a0e310f-9315-497a-bbd1-e6ce15a90119-tls-certs\") pod \"8a0e310f-9315-497a-bbd1-e6ce15a90119\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") "
Jul 20 10:14:00.420192 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.418924    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-tmp-dir\") pod \"8a0e310f-9315-497a-bbd1-e6ce15a90119\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") "
Jul 20 10:14:00.420192 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.418952    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjrdk\" (UniqueName: \"kubernetes.io/projected/8a0e310f-9315-497a-bbd1-e6ce15a90119-kube-api-access-fjrdk\") pod \"8a0e310f-9315-497a-bbd1-e6ce15a90119\" (UID: \"8a0e310f-9315-497a-bbd1-e6ce15a90119\") "
Jul 20 10:14:00.424675 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.424421    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a0e310f-9315-497a-bbd1-e6ce15a90119" (UID: "8a0e310f-9315-497a-bbd1-e6ce15a90119"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:14:00.424675 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.424611    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-home" (OuterVolumeSpecName: "home") pod "8a0e310f-9315-497a-bbd1-e6ce15a90119" (UID: "8a0e310f-9315-497a-bbd1-e6ce15a90119"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:14:00.424675 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.424624    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a0e310f-9315-497a-bbd1-e6ce15a90119" (UID: "8a0e310f-9315-497a-bbd1-e6ce15a90119"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:14:00.425677 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.425652    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-tmp-dir\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:14:00.425871 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.425827    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-home\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:14:00.426035 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.426023    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-model-cache\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:14:00.426375 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.426305    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0e310f-9315-497a-bbd1-e6ce15a90119-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a0e310f-9315-497a-bbd1-e6ce15a90119" (UID: "8a0e310f-9315-497a-bbd1-e6ce15a90119"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:14:00.427569 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.427525    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-dshm" (OuterVolumeSpecName: "dshm") pod "8a0e310f-9315-497a-bbd1-e6ce15a90119" (UID: "8a0e310f-9315-497a-bbd1-e6ce15a90119"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:14:00.429544 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.429514    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0e310f-9315-497a-bbd1-e6ce15a90119-kube-api-access-fjrdk" (OuterVolumeSpecName: "kube-api-access-fjrdk") pod "8a0e310f-9315-497a-bbd1-e6ce15a90119" (UID: "8a0e310f-9315-497a-bbd1-e6ce15a90119"). InnerVolumeSpecName "kube-api-access-fjrdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:14:00.527325 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.527288    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a0e310f-9315-497a-bbd1-e6ce15a90119-dshm\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:14:00.527325 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.527321    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a0e310f-9315-497a-bbd1-e6ce15a90119-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:14:00.527325 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:00.527332    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjrdk\" (UniqueName: \"kubernetes.io/projected/8a0e310f-9315-497a-bbd1-e6ce15a90119-kube-api-access-fjrdk\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:14:01.107736 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:01.107690    2580 generic.go:356] "Generic (PLEG): container finished" podID="8a0e310f-9315-497a-bbd1-e6ce15a90119" containerID="9f805eab6bad541f96eaac2e0cfa4dc53189ddf3107855d173f3ebba337bd42b" exitCode=0
Jul 20 10:14:01.107954 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:01.107856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw" event={"ID":"8a0e310f-9315-497a-bbd1-e6ce15a90119","Type":"ContainerDied","Data":"9f805eab6bad541f96eaac2e0cfa4dc53189ddf3107855d173f3ebba337bd42b"}
Jul 20 10:14:01.107954 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:01.107907    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw" event={"ID":"8a0e310f-9315-497a-bbd1-e6ce15a90119","Type":"ContainerDied","Data":"5651a1574f17745d8cffc03112c12ccf65f84621c2aa11d9e13d5d05d74cdf8d"}
Jul 20 10:14:01.107954 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:01.107938    2580 scope.go:117] "RemoveContainer" containerID="9f805eab6bad541f96eaac2e0cfa4dc53189ddf3107855d173f3ebba337bd42b"
Jul 20 10:14:01.108126 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:01.108108    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"
Jul 20 10:14:01.132739 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:01.132397    2580 scope.go:117] "RemoveContainer" containerID="9f805eab6bad541f96eaac2e0cfa4dc53189ddf3107855d173f3ebba337bd42b"
Jul 20 10:14:01.133055 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:14:01.132848    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f805eab6bad541f96eaac2e0cfa4dc53189ddf3107855d173f3ebba337bd42b\": container with ID starting with 9f805eab6bad541f96eaac2e0cfa4dc53189ddf3107855d173f3ebba337bd42b not found: ID does not exist" containerID="9f805eab6bad541f96eaac2e0cfa4dc53189ddf3107855d173f3ebba337bd42b"
Jul 20 10:14:01.133055 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:01.132885    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f805eab6bad541f96eaac2e0cfa4dc53189ddf3107855d173f3ebba337bd42b"} err="failed to get container status \"9f805eab6bad541f96eaac2e0cfa4dc53189ddf3107855d173f3ebba337bd42b\": rpc error: code = NotFound desc = could not find container \"9f805eab6bad541f96eaac2e0cfa4dc53189ddf3107855d173f3ebba337bd42b\": container with ID starting with 9f805eab6bad541f96eaac2e0cfa4dc53189ddf3107855d173f3ebba337bd42b not found: ID does not exist"
Jul 20 10:14:01.139211 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:01.139185    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"]
Jul 20 10:14:01.145788 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:01.145761    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfx94dw"]
Jul 20 10:14:02.415257 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:02.415212    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a0e310f-9315-497a-bbd1-e6ce15a90119" path="/var/lib/kubelet/pods/8a0e310f-9315-497a-bbd1-e6ce15a90119/volumes"
Jul 20 10:14:03.299571 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:03.299532    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:03.315186 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:03.315143    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"]
Jul 20 10:14:03.315404 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:03.315377    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg" podUID="19e0b47e-81f1-4139-941b-76d44d85927a" containerName="istio-proxy" containerID="cri-o://16fe77c702fdba917cd9e86c8c72fa82235ebed3b4e61d4a9e1c9fd904ca0806" gracePeriod=30
Jul 20 10:14:03.376499 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:03.376455    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:03.717534 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:03.717495    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:05.677308 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.677286    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:14:05.775816 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.775786    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-credential-socket\") pod \"19e0b47e-81f1-4139-941b-76d44d85927a\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") "
Jul 20 10:14:05.775991 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.775829    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e0b47e-81f1-4139-941b-76d44d85927a-istiod-ca-cert\") pod \"19e0b47e-81f1-4139-941b-76d44d85927a\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") "
Jul 20 10:14:05.775991 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.775852    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e0b47e-81f1-4139-941b-76d44d85927a-istio-token\") pod \"19e0b47e-81f1-4139-941b-76d44d85927a\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") "
Jul 20 10:14:05.775991 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.775881    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-workload-certs\") pod \"19e0b47e-81f1-4139-941b-76d44d85927a\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") "
Jul 20 10:14:05.776146 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.776056    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e0b47e-81f1-4139-941b-76d44d85927a-istio-podinfo\") pod \"19e0b47e-81f1-4139-941b-76d44d85927a\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") "
Jul 20 10:14:05.776146 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.776107    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-istio-envoy\") pod \"19e0b47e-81f1-4139-941b-76d44d85927a\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") "
Jul 20 10:14:05.776146 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.776132    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "19e0b47e-81f1-4139-941b-76d44d85927a" (UID: "19e0b47e-81f1-4139-941b-76d44d85927a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:14:05.776321 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.776147    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "19e0b47e-81f1-4139-941b-76d44d85927a" (UID: "19e0b47e-81f1-4139-941b-76d44d85927a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:14:05.776321 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.776158    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2sxq\" (UniqueName: \"kubernetes.io/projected/19e0b47e-81f1-4139-941b-76d44d85927a-kube-api-access-z2sxq\") pod \"19e0b47e-81f1-4139-941b-76d44d85927a\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") "
Jul 20 10:14:05.776321 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.776257    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-istio-data\") pod \"19e0b47e-81f1-4139-941b-76d44d85927a\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") "
Jul 20 10:14:05.776321 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.776297    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-workload-socket\") pod \"19e0b47e-81f1-4139-941b-76d44d85927a\" (UID: \"19e0b47e-81f1-4139-941b-76d44d85927a\") "
Jul 20 10:14:05.776321 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.776297    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e0b47e-81f1-4139-941b-76d44d85927a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "19e0b47e-81f1-4139-941b-76d44d85927a" (UID: "19e0b47e-81f1-4139-941b-76d44d85927a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:14:05.776589 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.776527    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-istio-data" (OuterVolumeSpecName: "istio-data") pod "19e0b47e-81f1-4139-941b-76d44d85927a" (UID: "19e0b47e-81f1-4139-941b-76d44d85927a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:14:05.776798 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.776732    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "19e0b47e-81f1-4139-941b-76d44d85927a" (UID: "19e0b47e-81f1-4139-941b-76d44d85927a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:14:05.776904 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.776889    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e0b47e-81f1-4139-941b-76d44d85927a-istiod-ca-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:14:05.776965 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.776910    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-workload-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:14:05.776965 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.776924    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-istio-data\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:14:05.776965 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.776937    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-workload-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:14:05.776965 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.776949    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-credential-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:14:05.778363 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.778331    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e0b47e-81f1-4139-941b-76d44d85927a-istio-token" (OuterVolumeSpecName: "istio-token") pod "19e0b47e-81f1-4139-941b-76d44d85927a" (UID: "19e0b47e-81f1-4139-941b-76d44d85927a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:14:05.778474 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.778423    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "19e0b47e-81f1-4139-941b-76d44d85927a" (UID: "19e0b47e-81f1-4139-941b-76d44d85927a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:14:05.778789 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.778765    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19e0b47e-81f1-4139-941b-76d44d85927a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "19e0b47e-81f1-4139-941b-76d44d85927a" (UID: "19e0b47e-81f1-4139-941b-76d44d85927a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 20 10:14:05.778860 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.778766    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e0b47e-81f1-4139-941b-76d44d85927a-kube-api-access-z2sxq" (OuterVolumeSpecName: "kube-api-access-z2sxq") pod "19e0b47e-81f1-4139-941b-76d44d85927a" (UID: "19e0b47e-81f1-4139-941b-76d44d85927a"). InnerVolumeSpecName "kube-api-access-z2sxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:14:05.878148 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.878115    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e0b47e-81f1-4139-941b-76d44d85927a-istio-token\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:14:05.878148 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.878147    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e0b47e-81f1-4139-941b-76d44d85927a-istio-podinfo\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:14:05.878148 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.878157    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e0b47e-81f1-4139-941b-76d44d85927a-istio-envoy\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:14:05.878418 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:05.878195    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2sxq\" (UniqueName: \"kubernetes.io/projected/19e0b47e-81f1-4139-941b-76d44d85927a-kube-api-access-z2sxq\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:14:06.140435 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:06.140327    2580 generic.go:356] "Generic (PLEG): container finished" podID="19e0b47e-81f1-4139-941b-76d44d85927a" containerID="16fe77c702fdba917cd9e86c8c72fa82235ebed3b4e61d4a9e1c9fd904ca0806" exitCode=0
Jul 20 10:14:06.140435 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:06.140385    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"
Jul 20 10:14:06.140435 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:06.140405    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg" event={"ID":"19e0b47e-81f1-4139-941b-76d44d85927a","Type":"ContainerDied","Data":"16fe77c702fdba917cd9e86c8c72fa82235ebed3b4e61d4a9e1c9fd904ca0806"}
Jul 20 10:14:06.140730 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:06.140466    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg" event={"ID":"19e0b47e-81f1-4139-941b-76d44d85927a","Type":"ContainerDied","Data":"af50de795c095de406a934bf7928cdd949bd1223982c84908b437d9af3aba019"}
Jul 20 10:14:06.140730 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:06.140497    2580 scope.go:117] "RemoveContainer" containerID="16fe77c702fdba917cd9e86c8c72fa82235ebed3b4e61d4a9e1c9fd904ca0806"
Jul 20 10:14:06.152632 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:06.152617    2580 scope.go:117] "RemoveContainer" containerID="16fe77c702fdba917cd9e86c8c72fa82235ebed3b4e61d4a9e1c9fd904ca0806"
Jul 20 10:14:06.152923 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:14:06.152905    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16fe77c702fdba917cd9e86c8c72fa82235ebed3b4e61d4a9e1c9fd904ca0806\": container with ID starting with 16fe77c702fdba917cd9e86c8c72fa82235ebed3b4e61d4a9e1c9fd904ca0806 not found: ID does not exist" containerID="16fe77c702fdba917cd9e86c8c72fa82235ebed3b4e61d4a9e1c9fd904ca0806"
Jul 20 10:14:06.153005 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:06.152931    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fe77c702fdba917cd9e86c8c72fa82235ebed3b4e61d4a9e1c9fd904ca0806"} err="failed to get container status \"16fe77c702fdba917cd9e86c8c72fa82235ebed3b4e61d4a9e1c9fd904ca0806\": rpc error: code = NotFound desc = could not find container \"16fe77c702fdba917cd9e86c8c72fa82235ebed3b4e61d4a9e1c9fd904ca0806\": container with ID starting with 16fe77c702fdba917cd9e86c8c72fa82235ebed3b4e61d4a9e1c9fd904ca0806 not found: ID does not exist"
Jul 20 10:14:06.165995 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:06.165970    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"]
Jul 20 10:14:06.172060 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:06.172038    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-84cf497d49-qwjsg"]
Jul 20 10:14:06.411125 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:06.411037    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19e0b47e-81f1-4139-941b-76d44d85927a" path="/var/lib/kubelet/pods/19e0b47e-81f1-4139-941b-76d44d85927a/volumes"
Jul 20 10:14:06.592663 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:06.592633    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:06.722895 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:06.722856    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:08.522184 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:08.522125    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:10.016797 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:10.016767    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:14.993079 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:14.993034    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:15.837718 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:15.837684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:15.959307 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:15.959278    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:17.526369 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:17.526337    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:19.911428 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:19.911344    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:20.010401 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:20.010360    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:26.084691 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:26.084655    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:26.225302 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:26.225263    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:30.638180 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.638121    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"]
Jul 20 10:14:30.638619 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.638469    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19e0b47e-81f1-4139-941b-76d44d85927a" containerName="istio-proxy"
Jul 20 10:14:30.638619 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.638481    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="19e0b47e-81f1-4139-941b-76d44d85927a" containerName="istio-proxy"
Jul 20 10:14:30.638619 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.638495    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0e310f-9315-497a-bbd1-e6ce15a90119" containerName="main"
Jul 20 10:14:30.638619 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.638500    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0e310f-9315-497a-bbd1-e6ce15a90119" containerName="main"
Jul 20 10:14:30.638619 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.638579    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a0e310f-9315-497a-bbd1-e6ce15a90119" containerName="main"
Jul 20 10:14:30.638619 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.638590    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="19e0b47e-81f1-4139-941b-76d44d85927a" containerName="istio-proxy"
Jul 20 10:14:30.641632 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.641615    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.644250 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.644228    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-fxpnp\""
Jul 20 10:14:30.645011 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.644988    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 20 10:14:30.645085 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.645013    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 20 10:14:30.645311 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.645296    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 20 10:14:30.655366 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.655344    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"]
Jul 20 10:14:30.697528 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.697495    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:30.797960 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.797924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m96z8\" (UniqueName: \"kubernetes.io/projected/795eca56-418a-4530-b4c8-ee9cc6a976bc-kube-api-access-m96z8\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.798119 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.797965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.798119 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.797997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.798119 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.798017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.798119 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.798111    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/795eca56-418a-4530-b4c8-ee9cc6a976bc-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.798290 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.798141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.898778 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.898694    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/795eca56-418a-4530-b4c8-ee9cc6a976bc-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.898778 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.898748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.898969 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.898798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m96z8\" (UniqueName: \"kubernetes.io/projected/795eca56-418a-4530-b4c8-ee9cc6a976bc-kube-api-access-m96z8\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.898969 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.898831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.898969 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.898859    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.899114 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.898990    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.899200 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.899145    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.899276 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.899212    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.899394 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.899373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.901105 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.901090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.901487 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.901472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/795eca56-418a-4530-b4c8-ee9cc6a976bc-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.910752 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.910731    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m96z8\" (UniqueName: \"kubernetes.io/projected/795eca56-418a-4530-b4c8-ee9cc6a976bc-kube-api-access-m96z8\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:30.931754 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.931720    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:30.951661 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:30.951631    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:31.103852 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:31.103821    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"]
Jul 20 10:14:31.106543 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:14:31.106518    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795eca56_418a_4530_b4c8_ee9cc6a976bc.slice/crio-86ee55f8e515b400dc69aa0086e4b7f1d820e4f53f297f9721bf141d906cd9b7 WatchSource:0}: Error finding container 86ee55f8e515b400dc69aa0086e4b7f1d820e4f53f297f9721bf141d906cd9b7: Status 404 returned error can't find the container with id 86ee55f8e515b400dc69aa0086e4b7f1d820e4f53f297f9721bf141d906cd9b7
Jul 20 10:14:31.266017 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:31.265981    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh" event={"ID":"795eca56-418a-4530-b4c8-ee9cc6a976bc","Type":"ContainerStarted","Data":"00353f44c6a4e9aa864aac93e4901e2bcfc11fc6864972e99a6195cb7a63e6e6"}
Jul 20 10:14:31.266214 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:31.266032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh" event={"ID":"795eca56-418a-4530-b4c8-ee9cc6a976bc","Type":"ContainerStarted","Data":"86ee55f8e515b400dc69aa0086e4b7f1d820e4f53f297f9721bf141d906cd9b7"}
Jul 20 10:14:31.288108 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:31.288056    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh" podStartSLOduration=1.28803025 podStartE2EDuration="1.28803025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:14:31.287022698 +0000 UTC m=+1083.335886346" watchObservedRunningTime="2026-07-20 10:14:31.28803025 +0000 UTC m=+1083.336893897"
Jul 20 10:14:31.952689 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:31.952650    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:33.994223 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:33.994193    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:39.498705 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:39.498664    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:40.150802 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:40.150766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:40.246687 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:40.246649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:40.951884 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:40.951842    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:40.965002 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:40.964980    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:14:41.331653 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:14:41.331571    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:15:11.369661 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.369621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:15:11.415045 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.415015    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"]
Jul 20 10:15:11.429741 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:15:11.429697    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:15:11.429741 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:15:11.429741 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:15:11.429741 ip-10-0-133-160 kubenswrapper[2580]:  > containerID="00353f44c6a4e9aa864aac93e4901e2bcfc11fc6864972e99a6195cb7a63e6e6" cmd=["/bin/sleep","15"]
Jul 20 10:15:11.430003 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.429751    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh" message="" err=<
Jul 20 10:15:11.430003 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:15:11.430003 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:15:11.430003 ip-10-0-133-160 kubenswrapper[2580]:  >
Jul 20 10:15:11.430003 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:15:11.429764    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:15:11.430003 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:15:11.430003 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:15:11.430003 ip-10-0-133-160 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh" podUID="795eca56-418a-4530-b4c8-ee9cc6a976bc" containerName="main" containerID="cri-o://00353f44c6a4e9aa864aac93e4901e2bcfc11fc6864972e99a6195cb7a63e6e6"
Jul 20 10:15:11.430003 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.429809    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh" podUID="795eca56-418a-4530-b4c8-ee9cc6a976bc" containerName="main" containerID="cri-o://00353f44c6a4e9aa864aac93e4901e2bcfc11fc6864972e99a6195cb7a63e6e6" gracePeriod=60
Jul 20 10:15:11.458561 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.458520    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:15:11.560532 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.560496    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:15:11.780728 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.780706    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:15:11.831456 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.831424    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-dshm\") pod \"795eca56-418a-4530-b4c8-ee9cc6a976bc\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") "
Jul 20 10:15:11.831456 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.831455    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-home\") pod \"795eca56-418a-4530-b4c8-ee9cc6a976bc\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") "
Jul 20 10:15:11.831685 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.831475    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-model-cache\") pod \"795eca56-418a-4530-b4c8-ee9cc6a976bc\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") "
Jul 20 10:15:11.831685 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.831507    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m96z8\" (UniqueName: \"kubernetes.io/projected/795eca56-418a-4530-b4c8-ee9cc6a976bc-kube-api-access-m96z8\") pod \"795eca56-418a-4530-b4c8-ee9cc6a976bc\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") "
Jul 20 10:15:11.831685 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.831537    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-tmp-dir\") pod \"795eca56-418a-4530-b4c8-ee9cc6a976bc\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") "
Jul 20 10:15:11.831685 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.831560    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/795eca56-418a-4530-b4c8-ee9cc6a976bc-tls-certs\") pod \"795eca56-418a-4530-b4c8-ee9cc6a976bc\" (UID: \"795eca56-418a-4530-b4c8-ee9cc6a976bc\") "
Jul 20 10:15:11.831888 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.831760    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-home" (OuterVolumeSpecName: "home") pod "795eca56-418a-4530-b4c8-ee9cc6a976bc" (UID: "795eca56-418a-4530-b4c8-ee9cc6a976bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:15:11.831888 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.831775    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "795eca56-418a-4530-b4c8-ee9cc6a976bc" (UID: "795eca56-418a-4530-b4c8-ee9cc6a976bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:15:11.831888 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.831844    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-home\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:15:11.831888 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.831861    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-model-cache\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:15:11.832075 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.831958    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "795eca56-418a-4530-b4c8-ee9cc6a976bc" (UID: "795eca56-418a-4530-b4c8-ee9cc6a976bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:15:11.833930 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.833899    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/795eca56-418a-4530-b4c8-ee9cc6a976bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "795eca56-418a-4530-b4c8-ee9cc6a976bc" (UID: "795eca56-418a-4530-b4c8-ee9cc6a976bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:15:11.833930 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.833909    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795eca56-418a-4530-b4c8-ee9cc6a976bc-kube-api-access-m96z8" (OuterVolumeSpecName: "kube-api-access-m96z8") pod "795eca56-418a-4530-b4c8-ee9cc6a976bc" (UID: "795eca56-418a-4530-b4c8-ee9cc6a976bc"). InnerVolumeSpecName "kube-api-access-m96z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:15:11.833930 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.833937    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-dshm" (OuterVolumeSpecName: "dshm") pod "795eca56-418a-4530-b4c8-ee9cc6a976bc" (UID: "795eca56-418a-4530-b4c8-ee9cc6a976bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:15:11.933074 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.932992    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m96z8\" (UniqueName: \"kubernetes.io/projected/795eca56-418a-4530-b4c8-ee9cc6a976bc-kube-api-access-m96z8\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:15:11.933074 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.933020    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-tmp-dir\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:15:11.933074 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.933032    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/795eca56-418a-4530-b4c8-ee9cc6a976bc-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:15:11.933074 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:11.933039    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/795eca56-418a-4530-b4c8-ee9cc6a976bc-dshm\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:15:12.484736 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:12.484704    2580 generic.go:356] "Generic (PLEG): container finished" podID="795eca56-418a-4530-b4c8-ee9cc6a976bc" containerID="00353f44c6a4e9aa864aac93e4901e2bcfc11fc6864972e99a6195cb7a63e6e6" exitCode=0
Jul 20 10:15:12.485193 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:12.484745    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"
Jul 20 10:15:12.485193 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:12.484798    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh" event={"ID":"795eca56-418a-4530-b4c8-ee9cc6a976bc","Type":"ContainerDied","Data":"00353f44c6a4e9aa864aac93e4901e2bcfc11fc6864972e99a6195cb7a63e6e6"}
Jul 20 10:15:12.485193 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:12.484849    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh" event={"ID":"795eca56-418a-4530-b4c8-ee9cc6a976bc","Type":"ContainerDied","Data":"86ee55f8e515b400dc69aa0086e4b7f1d820e4f53f297f9721bf141d906cd9b7"}
Jul 20 10:15:12.485193 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:12.484872    2580 scope.go:117] "RemoveContainer" containerID="00353f44c6a4e9aa864aac93e4901e2bcfc11fc6864972e99a6195cb7a63e6e6"
Jul 20 10:15:12.494004 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:12.493989    2580 scope.go:117] "RemoveContainer" containerID="00353f44c6a4e9aa864aac93e4901e2bcfc11fc6864972e99a6195cb7a63e6e6"
Jul 20 10:15:12.494319 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:15:12.494290    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00353f44c6a4e9aa864aac93e4901e2bcfc11fc6864972e99a6195cb7a63e6e6\": container with ID starting with 00353f44c6a4e9aa864aac93e4901e2bcfc11fc6864972e99a6195cb7a63e6e6 not found: ID does not exist" containerID="00353f44c6a4e9aa864aac93e4901e2bcfc11fc6864972e99a6195cb7a63e6e6"
Jul 20 10:15:12.494376 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:12.494328    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00353f44c6a4e9aa864aac93e4901e2bcfc11fc6864972e99a6195cb7a63e6e6"} err="failed to get container status \"00353f44c6a4e9aa864aac93e4901e2bcfc11fc6864972e99a6195cb7a63e6e6\": rpc error: code = NotFound desc = could not find container \"00353f44c6a4e9aa864aac93e4901e2bcfc11fc6864972e99a6195cb7a63e6e6\": container with ID starting with 00353f44c6a4e9aa864aac93e4901e2bcfc11fc6864972e99a6195cb7a63e6e6 not found: ID does not exist"
Jul 20 10:15:12.507581 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:12.507560    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"]
Jul 20 10:15:12.519069 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:12.519049    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58f7rdnh"]
Jul 20 10:15:14.411100 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:14.411068    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="795eca56-418a-4530-b4c8-ee9cc6a976bc" path="/var/lib/kubelet/pods/795eca56-418a-4530-b4c8-ee9cc6a976bc/volumes"
Jul 20 10:15:31.067213 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:31.067148    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:15:31.160537 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:31.160508    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:15:38.044656 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.044614    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:15:38.670014 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.669980    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"]
Jul 20 10:15:38.670326 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.670314    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795eca56-418a-4530-b4c8-ee9cc6a976bc" containerName="main"
Jul 20 10:15:38.670394 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.670328    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="795eca56-418a-4530-b4c8-ee9cc6a976bc" containerName="main"
Jul 20 10:15:38.670394 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.670390    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="795eca56-418a-4530-b4c8-ee9cc6a976bc" containerName="main"
Jul 20 10:15:38.674473 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.674454    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.678924 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.678903    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 20 10:15:38.679573 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.679557    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 20 10:15:38.679573 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.679566    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-9sptf\""
Jul 20 10:15:38.679718 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.679601    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 20 10:15:38.679948 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.679929    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 20 10:15:38.699622 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.699591    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"]
Jul 20 10:15:38.717499 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.717454    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:15:38.767564 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.767530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-dshm\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.767766 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.767573    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.767766 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.767605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e95c3dd-8576-402b-b035-5b8e8cc1126d-tls-certs\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.767766 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.767684    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-home\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.767766 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.767713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-model-cache\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.767766 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.767732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqn6k\" (UniqueName: \"kubernetes.io/projected/6e95c3dd-8576-402b-b035-5b8e8cc1126d-kube-api-access-dqn6k\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.768048 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.767794    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-tmp-dir\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.869235 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.869193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-dshm\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.869412 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.869248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.869412 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.869282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e95c3dd-8576-402b-b035-5b8e8cc1126d-tls-certs\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.869412 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.869330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-home\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.869412 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.869354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-model-cache\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.869412 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.869380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqn6k\" (UniqueName: \"kubernetes.io/projected/6e95c3dd-8576-402b-b035-5b8e8cc1126d-kube-api-access-dqn6k\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.869664 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.869415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-tmp-dir\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.869725 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.869696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.869781 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.869760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-model-cache\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.869841 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.869819    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-tmp-dir\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.869874 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.869851    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-home\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.871661 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.871639    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-dshm\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.871808 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.871792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e95c3dd-8576-402b-b035-5b8e8cc1126d-tls-certs\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.881762 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.881734    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqn6k\" (UniqueName: \"kubernetes.io/projected/6e95c3dd-8576-402b-b035-5b8e8cc1126d-kube-api-access-dqn6k\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-55k9d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:38.984586 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:38.984543    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:39.023788 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:39.023744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:15:39.144632 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:39.144609    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"]
Jul 20 10:15:39.146949 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:15:39.146924    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e95c3dd_8576_402b_b035_5b8e8cc1126d.slice/crio-ee3120099acdf5c4a02cef52dce8c4a576b3d688c0d28919f715714550e76bcb WatchSource:0}: Error finding container ee3120099acdf5c4a02cef52dce8c4a576b3d688c0d28919f715714550e76bcb: Status 404 returned error can't find the container with id ee3120099acdf5c4a02cef52dce8c4a576b3d688c0d28919f715714550e76bcb
Jul 20 10:15:39.148853 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:39.148837    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:15:39.627963 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:39.627934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d" event={"ID":"6e95c3dd-8576-402b-b035-5b8e8cc1126d","Type":"ContainerStarted","Data":"ee3120099acdf5c4a02cef52dce8c4a576b3d688c0d28919f715714550e76bcb"}
Jul 20 10:15:43.661083 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:43.661031    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d" event={"ID":"6e95c3dd-8576-402b-b035-5b8e8cc1126d","Type":"ContainerStarted","Data":"b9e8b3682456ea6b163060112d2e189baff2717c37ad625e5624eeb876dade34"}
Jul 20 10:15:44.185010 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:44.184966    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:15:44.262754 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:44.262725    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:15:46.679493 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:46.679453    2580 generic.go:356] "Generic (PLEG): container finished" podID="6e95c3dd-8576-402b-b035-5b8e8cc1126d" containerID="b9e8b3682456ea6b163060112d2e189baff2717c37ad625e5624eeb876dade34" exitCode=0
Jul 20 10:15:46.679902 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:46.679528    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d" event={"ID":"6e95c3dd-8576-402b-b035-5b8e8cc1126d","Type":"ContainerDied","Data":"b9e8b3682456ea6b163060112d2e189baff2717c37ad625e5624eeb876dade34"}
Jul 20 10:15:47.686028 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:47.685995    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d" event={"ID":"6e95c3dd-8576-402b-b035-5b8e8cc1126d","Type":"ContainerStarted","Data":"71d2cf4c8df2d973f09177e02f22fd1ed94303632ae6dc9f682681946b011b31"}
Jul 20 10:15:47.710523 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:47.710466    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d" podStartSLOduration=3.020133728 podStartE2EDuration="9.71045111s" totalImagesPullingTime="3.531193583s" totalInitContainerRuntime="3.159123799s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:15:47.708090619 +0000 UTC m=+1159.756954303" watchObservedRunningTime="2026-07-20 10:15:47.71045111 +0000 UTC m=+1159.759314757"
Jul 20 10:15:47.984879 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:47.984850    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:48.984912 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:48.984860    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:48.997705 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:48.997675    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:15:49.555610 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:49.555562    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:15:49.706798 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:15:49.706760    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:16:22.293488 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:16:22.293447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:16:31.564154 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:16:31.564110    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:17:11.891047 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:11.891013    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"]
Jul 20 10:17:11.900527 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:17:11.900476    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:17:11.900527 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:17:11.900527 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:17:11.900527 ip-10-0-133-160 kubenswrapper[2580]:  > containerID="71d2cf4c8df2d973f09177e02f22fd1ed94303632ae6dc9f682681946b011b31" cmd=["/bin/sleep","15"]
Jul 20 10:17:11.900857 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:11.900559    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d" message="" err=<
Jul 20 10:17:11.900857 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:17:11.900857 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:17:11.900857 ip-10-0-133-160 kubenswrapper[2580]:  >
Jul 20 10:17:11.900857 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:17:11.900576    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:17:11.900857 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:17:11.900857 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:17:11.900857 ip-10-0-133-160 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d" podUID="6e95c3dd-8576-402b-b035-5b8e8cc1126d" containerName="main" containerID="cri-o://71d2cf4c8df2d973f09177e02f22fd1ed94303632ae6dc9f682681946b011b31"
Jul 20 10:17:11.900857 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:11.900633    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d" podUID="6e95c3dd-8576-402b-b035-5b8e8cc1126d" containerName="main" containerID="cri-o://71d2cf4c8df2d973f09177e02f22fd1ed94303632ae6dc9f682681946b011b31" gracePeriod=60
Jul 20 10:17:11.901719 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:11.901694    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:17:11.970463 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:11.970427    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:17:12.117132 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.117100    2580 generic.go:356] "Generic (PLEG): container finished" podID="6e95c3dd-8576-402b-b035-5b8e8cc1126d" containerID="71d2cf4c8df2d973f09177e02f22fd1ed94303632ae6dc9f682681946b011b31" exitCode=0
Jul 20 10:17:12.117329 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.117187    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d" event={"ID":"6e95c3dd-8576-402b-b035-5b8e8cc1126d","Type":"ContainerDied","Data":"71d2cf4c8df2d973f09177e02f22fd1ed94303632ae6dc9f682681946b011b31"}
Jul 20 10:17:12.151397 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.151336    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:17:12.244113 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.244076    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-tmp-dir\") pod \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") "
Jul 20 10:17:12.244333 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.244123    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqn6k\" (UniqueName: \"kubernetes.io/projected/6e95c3dd-8576-402b-b035-5b8e8cc1126d-kube-api-access-dqn6k\") pod \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") "
Jul 20 10:17:12.244333 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.244146    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-home\") pod \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") "
Jul 20 10:17:12.244333 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.244246    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-dshm\") pod \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") "
Jul 20 10:17:12.244333 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.244294    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e95c3dd-8576-402b-b035-5b8e8cc1126d-tls-certs\") pod \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") "
Jul 20 10:17:12.244561 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.244335    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-kserve-provision-location\") pod \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") "
Jul 20 10:17:12.244561 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.244361    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-model-cache\") pod \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\" (UID: \"6e95c3dd-8576-402b-b035-5b8e8cc1126d\") "
Jul 20 10:17:12.244561 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.244421    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-home" (OuterVolumeSpecName: "home") pod "6e95c3dd-8576-402b-b035-5b8e8cc1126d" (UID: "6e95c3dd-8576-402b-b035-5b8e8cc1126d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:17:12.244561 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.244442    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e95c3dd-8576-402b-b035-5b8e8cc1126d" (UID: "6e95c3dd-8576-402b-b035-5b8e8cc1126d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:17:12.244744 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.244642    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-tmp-dir\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:17:12.244744 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.244661    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-home\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:17:12.244811 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.244736    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e95c3dd-8576-402b-b035-5b8e8cc1126d" (UID: "6e95c3dd-8576-402b-b035-5b8e8cc1126d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:17:12.246652 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.246619    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e95c3dd-8576-402b-b035-5b8e8cc1126d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e95c3dd-8576-402b-b035-5b8e8cc1126d" (UID: "6e95c3dd-8576-402b-b035-5b8e8cc1126d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:17:12.246652 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.246645    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e95c3dd-8576-402b-b035-5b8e8cc1126d-kube-api-access-dqn6k" (OuterVolumeSpecName: "kube-api-access-dqn6k") pod "6e95c3dd-8576-402b-b035-5b8e8cc1126d" (UID: "6e95c3dd-8576-402b-b035-5b8e8cc1126d"). InnerVolumeSpecName "kube-api-access-dqn6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:17:12.246828 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.246726    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-dshm" (OuterVolumeSpecName: "dshm") pod "6e95c3dd-8576-402b-b035-5b8e8cc1126d" (UID: "6e95c3dd-8576-402b-b035-5b8e8cc1126d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:17:12.298954 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.298907    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e95c3dd-8576-402b-b035-5b8e8cc1126d" (UID: "6e95c3dd-8576-402b-b035-5b8e8cc1126d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:17:12.345294 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.345257    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-dshm\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:17:12.345294 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.345287    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e95c3dd-8576-402b-b035-5b8e8cc1126d-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:17:12.345294 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.345301    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-kserve-provision-location\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:17:12.345523 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.345312    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e95c3dd-8576-402b-b035-5b8e8cc1126d-model-cache\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:17:12.345523 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:12.345327    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqn6k\" (UniqueName: \"kubernetes.io/projected/6e95c3dd-8576-402b-b035-5b8e8cc1126d-kube-api-access-dqn6k\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:17:13.123909 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:13.123874    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d" event={"ID":"6e95c3dd-8576-402b-b035-5b8e8cc1126d","Type":"ContainerDied","Data":"ee3120099acdf5c4a02cef52dce8c4a576b3d688c0d28919f715714550e76bcb"}
Jul 20 10:17:13.123909 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:13.123892    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"
Jul 20 10:17:13.124527 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:13.123972    2580 scope.go:117] "RemoveContainer" containerID="71d2cf4c8df2d973f09177e02f22fd1ed94303632ae6dc9f682681946b011b31"
Jul 20 10:17:13.133280 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:13.133261    2580 scope.go:117] "RemoveContainer" containerID="b9e8b3682456ea6b163060112d2e189baff2717c37ad625e5624eeb876dade34"
Jul 20 10:17:13.141289 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:13.141265    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"]
Jul 20 10:17:13.145891 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:13.145866    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-55k9d"]
Jul 20 10:17:14.410910 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:14.410879    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e95c3dd-8576-402b-b035-5b8e8cc1126d" path="/var/lib/kubelet/pods/6e95c3dd-8576-402b-b035-5b8e8cc1126d/volumes"
Jul 20 10:17:31.087265 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:31.087227    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:17:31.891011 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:31.890974    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:17:31.977224 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:31.977192    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:17:34.812181 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:34.812124    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:17:38.825619 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:38.825577    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:17:41.535287 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:41.535248    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:17:41.797687 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:41.797593    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:17:45.128986 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:45.128948    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:17:45.236924 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:45.236887    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:17:47.205415 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:47.205387    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:17:48.559232 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:48.559199    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:17:48.642406 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:48.642372    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:17:49.094704 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:17:49.094667    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:18:15.452193 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:15.452117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:18:16.636805 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:16.636770    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:18:35.399284 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:35.399247    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:18:35.483411 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:35.483374    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:18:38.354079 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:38.354049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:18:38.451526 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:38.451467    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:18:38.978362 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:38.978326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:18:39.517344 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:39.517308    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:18:42.591117 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:42.591081    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:18:43.412064 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:43.412018    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:18:48.676661 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:48.676628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:18:48.764032 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:48.763997    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:18:58.587900 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.587865    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:18:58.905177 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.905078    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"]
Jul 20 10:18:58.905459 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.905445    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e95c3dd-8576-402b-b035-5b8e8cc1126d" containerName="storage-initializer"
Jul 20 10:18:58.905459 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.905460    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e95c3dd-8576-402b-b035-5b8e8cc1126d" containerName="storage-initializer"
Jul 20 10:18:58.905541 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.905469    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e95c3dd-8576-402b-b035-5b8e8cc1126d" containerName="main"
Jul 20 10:18:58.905541 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.905475    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e95c3dd-8576-402b-b035-5b8e8cc1126d" containerName="main"
Jul 20 10:18:58.905541 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.905535    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e95c3dd-8576-402b-b035-5b8e8cc1126d" containerName="main"
Jul 20 10:18:58.908606 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.908588    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"
Jul 20 10:18:58.913317 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.913270    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Jul 20 10:18:58.913439 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.913286    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-h267p\""
Jul 20 10:18:58.913439 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.913411    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-fpvr9\""
Jul 20 10:18:58.915778 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.915759    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 20 10:18:58.915915 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.915895    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Jul 20 10:18:58.917127 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.917109    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Jul 20 10:18:58.922017 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.921995    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c559d7ef-c3c1-420e-91ed-79922eb9cb65-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh\" (UID: \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"
Jul 20 10:18:58.922110 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.922025    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47s9w\" (UniqueName: \"kubernetes.io/projected/c559d7ef-c3c1-420e-91ed-79922eb9cb65-kube-api-access-47s9w\") pod \"auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh\" (UID: \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"
Jul 20 10:18:58.922110 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.922050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c559d7ef-c3c1-420e-91ed-79922eb9cb65-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh\" (UID: \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"
Jul 20 10:18:58.922110 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.922066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c559d7ef-c3c1-420e-91ed-79922eb9cb65-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh\" (UID: \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"
Jul 20 10:18:58.932578 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.932542    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"]
Jul 20 10:18:58.981253 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:58.981207    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:18:59.023517 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:59.023480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c559d7ef-c3c1-420e-91ed-79922eb9cb65-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh\" (UID: \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"
Jul 20 10:18:59.023517 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:59.023517    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47s9w\" (UniqueName: \"kubernetes.io/projected/c559d7ef-c3c1-420e-91ed-79922eb9cb65-kube-api-access-47s9w\") pod \"auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh\" (UID: \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"
Jul 20 10:18:59.023761 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:59.023542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c559d7ef-c3c1-420e-91ed-79922eb9cb65-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh\" (UID: \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"
Jul 20 10:18:59.023761 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:59.023658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c559d7ef-c3c1-420e-91ed-79922eb9cb65-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh\" (UID: \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"
Jul 20 10:18:59.024050 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:59.024031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c559d7ef-c3c1-420e-91ed-79922eb9cb65-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh\" (UID: \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"
Jul 20 10:18:59.024109 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:59.024048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c559d7ef-c3c1-420e-91ed-79922eb9cb65-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh\" (UID: \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"
Jul 20 10:18:59.026201 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:59.026162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c559d7ef-c3c1-420e-91ed-79922eb9cb65-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh\" (UID: \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"
Jul 20 10:18:59.033154 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:59.033129    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47s9w\" (UniqueName: \"kubernetes.io/projected/c559d7ef-c3c1-420e-91ed-79922eb9cb65-kube-api-access-47s9w\") pod \"auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh\" (UID: \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"
Jul 20 10:18:59.218952 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:59.218916    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"
Jul 20 10:18:59.366837 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:59.366781    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"]
Jul 20 10:18:59.369213 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:18:59.369183    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc559d7ef_c3c1_420e_91ed_79922eb9cb65.slice/crio-1bf455ac974a606d970f8a668ffa6955a982fd4e50afc962cfb08706e83b25da WatchSource:0}: Error finding container 1bf455ac974a606d970f8a668ffa6955a982fd4e50afc962cfb08706e83b25da: Status 404 returned error can't find the container with id 1bf455ac974a606d970f8a668ffa6955a982fd4e50afc962cfb08706e83b25da
Jul 20 10:18:59.655212 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:59.655107    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh" event={"ID":"c559d7ef-c3c1-420e-91ed-79922eb9cb65","Type":"ContainerStarted","Data":"4e9a3cd4ef5a9d5985dda17ef8845e2b380ffe1b292899b3aad1ee09b7b5adc0"}
Jul 20 10:18:59.655212 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:18:59.655185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh" event={"ID":"c559d7ef-c3c1-420e-91ed-79922eb9cb65","Type":"ContainerStarted","Data":"1bf455ac974a606d970f8a668ffa6955a982fd4e50afc962cfb08706e83b25da"}
Jul 20 10:19:00.662290 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:00.662255    2580 generic.go:356] "Generic (PLEG): container finished" podID="c559d7ef-c3c1-420e-91ed-79922eb9cb65" containerID="4e9a3cd4ef5a9d5985dda17ef8845e2b380ffe1b292899b3aad1ee09b7b5adc0" exitCode=0
Jul 20 10:19:00.662794 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:00.662309    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh" event={"ID":"c559d7ef-c3c1-420e-91ed-79922eb9cb65","Type":"ContainerDied","Data":"4e9a3cd4ef5a9d5985dda17ef8845e2b380ffe1b292899b3aad1ee09b7b5adc0"}
Jul 20 10:19:03.684928 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:03.684833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh" event={"ID":"c559d7ef-c3c1-420e-91ed-79922eb9cb65","Type":"ContainerStarted","Data":"db30d9da83652aad44d543a92ba4e6d9282faa8c59d45116a6de8de1689e6d54"}
Jul 20 10:19:03.685382 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:03.684931    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"
Jul 20 10:19:03.706932 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:03.706883    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh" podStartSLOduration=2.446500502 podStartE2EDuration="5.706866502s" totalImagesPullingTime="2.593389203s" totalInitContainerRuntime="666.976797ms" isStatefulPod=false podCreationTimestamp="2026-07-20 10:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:19:03.704859106 +0000 UTC m=+1355.753722753" watchObservedRunningTime="2026-07-20 10:19:03.706866502 +0000 UTC m=+1355.755730149"
Jul 20 10:19:04.513233 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:04.513196    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:19:16.728839 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:16.728781    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:19:20.445566 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:20.445532    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:19:20.544063 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:20.544031    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:19:21.841236 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:21.841198    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:19:34.692790 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:34.692762    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"
Jul 20 10:19:37.199615 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.199577    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"]
Jul 20 10:19:37.204632 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.204609    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.206877 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.206854    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 20 10:19:37.207583 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.207564    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 20 10:19:37.207699 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.207586    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 20 10:19:37.207699 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.207565    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-mx7hf\""
Jul 20 10:19:37.217856 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.217829    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"]
Jul 20 10:19:37.249552 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.249514    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:19:37.351453 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.351414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.351619 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.351482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.351619 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.351536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.351619 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.351564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.351729 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.351628    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tfwq\" (UniqueName: \"kubernetes.io/projected/945d1d32-4f7d-4da2-9304-2413fc86b0dd-kube-api-access-8tfwq\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.351729 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.351669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/945d1d32-4f7d-4da2-9304-2413fc86b0dd-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.452294 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.452259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.452477 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.452314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tfwq\" (UniqueName: \"kubernetes.io/projected/945d1d32-4f7d-4da2-9304-2413fc86b0dd-kube-api-access-8tfwq\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.452477 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.452347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/945d1d32-4f7d-4da2-9304-2413fc86b0dd-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.452477 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.452374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.452477 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.452409    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.452711 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.452589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.452842 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.452822    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.452920 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.452859    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.452920 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.452887    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.454749 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.454726    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.454908 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.454891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/945d1d32-4f7d-4da2-9304-2413fc86b0dd-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.471359 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.471334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tfwq\" (UniqueName: \"kubernetes.io/projected/945d1d32-4f7d-4da2-9304-2413fc86b0dd-kube-api-access-8tfwq\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.483346 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.483318    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:19:37.518281 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.518246    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:37.668371 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.668297    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"]
Jul 20 10:19:37.671198 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:19:37.671149    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945d1d32_4f7d_4da2_9304_2413fc86b0dd.slice/crio-c9fc3720b0d2d42674ab08eecc75455356d0c315a90fd0d29af4d3e31cb25555 WatchSource:0}: Error finding container c9fc3720b0d2d42674ab08eecc75455356d0c315a90fd0d29af4d3e31cb25555: Status 404 returned error can't find the container with id c9fc3720b0d2d42674ab08eecc75455356d0c315a90fd0d29af4d3e31cb25555
Jul 20 10:19:37.861627 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.861591    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz" event={"ID":"945d1d32-4f7d-4da2-9304-2413fc86b0dd","Type":"ContainerStarted","Data":"b002fd602610a29952b54cefe849113ba07fcb3b994ab3cb540a60330550df88"}
Jul 20 10:19:37.861790 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.861644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz" event={"ID":"945d1d32-4f7d-4da2-9304-2413fc86b0dd","Type":"ContainerStarted","Data":"c9fc3720b0d2d42674ab08eecc75455356d0c315a90fd0d29af4d3e31cb25555"}
Jul 20 10:19:37.882234 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:37.882158    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz" podStartSLOduration=0.88213646 podStartE2EDuration="882.13646ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:19:37.88120327 +0000 UTC m=+1389.930066913" watchObservedRunningTime="2026-07-20 10:19:37.88213646 +0000 UTC m=+1389.931000101"
Jul 20 10:19:38.518842 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:38.518792    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:45.531646 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:45.531618    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:19:45.631505 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:45.631466    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:19:47.519293 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:47.519261    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:47.532749 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:47.532721    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:47.928679 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:47.928596    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:19:50.150424 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:19:50.150385    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:10.546873 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:10.546835    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:10.554726 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:10.554698    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"]
Jul 20 10:20:10.564420 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:20:10.564388    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:20:10.564420 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:20:10.564420 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:20:10.564420 ip-10-0-133-160 kubenswrapper[2580]:  > containerID="b002fd602610a29952b54cefe849113ba07fcb3b994ab3cb540a60330550df88" cmd=["/bin/sleep","15"]
Jul 20 10:20:10.564714 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:10.564444    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz" message="" err=<
Jul 20 10:20:10.564714 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:20:10.564714 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:20:10.564714 ip-10-0-133-160 kubenswrapper[2580]:  >
Jul 20 10:20:10.564714 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:20:10.564463    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:20:10.564714 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:20:10.564714 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:20:10.564714 ip-10-0-133-160 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz" podUID="945d1d32-4f7d-4da2-9304-2413fc86b0dd" containerName="main" containerID="cri-o://b002fd602610a29952b54cefe849113ba07fcb3b994ab3cb540a60330550df88"
Jul 20 10:20:10.564714 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:10.564506    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz" podUID="945d1d32-4f7d-4da2-9304-2413fc86b0dd" containerName="main" containerID="cri-o://b002fd602610a29952b54cefe849113ba07fcb3b994ab3cb540a60330550df88" gracePeriod=60
Jul 20 10:20:10.617389 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:10.617355    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:10.918278 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:10.918154    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz" podUID="945d1d32-4f7d-4da2-9304-2413fc86b0dd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 20 10:20:11.041266 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.041231    2580 generic.go:356] "Generic (PLEG): container finished" podID="945d1d32-4f7d-4da2-9304-2413fc86b0dd" containerID="b002fd602610a29952b54cefe849113ba07fcb3b994ab3cb540a60330550df88" exitCode=0
Jul 20 10:20:11.041428 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.041299    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz" event={"ID":"945d1d32-4f7d-4da2-9304-2413fc86b0dd","Type":"ContainerDied","Data":"b002fd602610a29952b54cefe849113ba07fcb3b994ab3cb540a60330550df88"}
Jul 20 10:20:11.313984 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.313959    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:20:11.440603 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.440504    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-home\") pod \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") "
Jul 20 10:20:11.440603 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.440547    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/945d1d32-4f7d-4da2-9304-2413fc86b0dd-tls-certs\") pod \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") "
Jul 20 10:20:11.440603 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.440593    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-tmp-dir\") pod \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") "
Jul 20 10:20:11.440902 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.440628    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-dshm\") pod \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") "
Jul 20 10:20:11.440902 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.440662    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-model-cache\") pod \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") "
Jul 20 10:20:11.440902 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.440687    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tfwq\" (UniqueName: \"kubernetes.io/projected/945d1d32-4f7d-4da2-9304-2413fc86b0dd-kube-api-access-8tfwq\") pod \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\" (UID: \"945d1d32-4f7d-4da2-9304-2413fc86b0dd\") "
Jul 20 10:20:11.440902 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.440850    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-home" (OuterVolumeSpecName: "home") pod "945d1d32-4f7d-4da2-9304-2413fc86b0dd" (UID: "945d1d32-4f7d-4da2-9304-2413fc86b0dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:20:11.441104 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.440914    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "945d1d32-4f7d-4da2-9304-2413fc86b0dd" (UID: "945d1d32-4f7d-4da2-9304-2413fc86b0dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:20:11.441104 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.440926    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "945d1d32-4f7d-4da2-9304-2413fc86b0dd" (UID: "945d1d32-4f7d-4da2-9304-2413fc86b0dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:20:11.441104 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.441010    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-model-cache\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:20:11.441104 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.441029    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-home\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:20:11.441104 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.441058    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-tmp-dir\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:20:11.443093 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.443065    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/945d1d32-4f7d-4da2-9304-2413fc86b0dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "945d1d32-4f7d-4da2-9304-2413fc86b0dd" (UID: "945d1d32-4f7d-4da2-9304-2413fc86b0dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:20:11.443225 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.443091    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-dshm" (OuterVolumeSpecName: "dshm") pod "945d1d32-4f7d-4da2-9304-2413fc86b0dd" (UID: "945d1d32-4f7d-4da2-9304-2413fc86b0dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:20:11.443225 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.443105    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945d1d32-4f7d-4da2-9304-2413fc86b0dd-kube-api-access-8tfwq" (OuterVolumeSpecName: "kube-api-access-8tfwq") pod "945d1d32-4f7d-4da2-9304-2413fc86b0dd" (UID: "945d1d32-4f7d-4da2-9304-2413fc86b0dd"). InnerVolumeSpecName "kube-api-access-8tfwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:20:11.542422 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.542382    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tfwq\" (UniqueName: \"kubernetes.io/projected/945d1d32-4f7d-4da2-9304-2413fc86b0dd-kube-api-access-8tfwq\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:20:11.542422 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.542412    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/945d1d32-4f7d-4da2-9304-2413fc86b0dd-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:20:11.542422 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:11.542423    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/945d1d32-4f7d-4da2-9304-2413fc86b0dd-dshm\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:20:12.048113 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:12.048083    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"
Jul 20 10:20:12.048577 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:12.048112    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz" event={"ID":"945d1d32-4f7d-4da2-9304-2413fc86b0dd","Type":"ContainerDied","Data":"c9fc3720b0d2d42674ab08eecc75455356d0c315a90fd0d29af4d3e31cb25555"}
Jul 20 10:20:12.048577 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:12.048213    2580 scope.go:117] "RemoveContainer" containerID="b002fd602610a29952b54cefe849113ba07fcb3b994ab3cb540a60330550df88"
Jul 20 10:20:12.070708 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:12.070686    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"]
Jul 20 10:20:12.075117 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:12.075094    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995m5vhz"]
Jul 20 10:20:12.411750 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:12.411662    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="945d1d32-4f7d-4da2-9304-2413fc86b0dd" path="/var/lib/kubelet/pods/945d1d32-4f7d-4da2-9304-2413fc86b0dd/volumes"
Jul 20 10:20:30.333255 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:30.333217    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:30.389139 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:30.389104    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:50.399023 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:50.398968    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:50.498651 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:50.498622    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:57.182301 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:57.182261    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:58.936991 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:58.936960    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:59.343328 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:20:59.343291    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:03.225415 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:03.225380    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:03.339224 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:03.339183    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:10.665751 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:10.665717    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:11.936843 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:11.936798    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:11.938113 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:11.938087    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"]
Jul 20 10:21:12.068633 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:12.068591    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:14.691788 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:14.691744    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh" podUID="c559d7ef-c3c1-420e-91ed-79922eb9cb65" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:21:24.692137 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:24.692099    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh" podUID="c559d7ef-c3c1-420e-91ed-79922eb9cb65" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:21:26.959340 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:26.959285    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh" podUID="c559d7ef-c3c1-420e-91ed-79922eb9cb65" containerName="main" containerID="cri-o://db30d9da83652aad44d543a92ba4e6d9282faa8c59d45116a6de8de1689e6d54" gracePeriod=45
Jul 20 10:21:27.209043 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.209019    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"
Jul 20 10:21:27.391808 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.391702    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c559d7ef-c3c1-420e-91ed-79922eb9cb65-tls-certs\") pod \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\" (UID: \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\") "
Jul 20 10:21:27.391808 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.391766    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c559d7ef-c3c1-420e-91ed-79922eb9cb65-tokenizer-uds\") pod \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\" (UID: \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\") "
Jul 20 10:21:27.391808 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.391797    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47s9w\" (UniqueName: \"kubernetes.io/projected/c559d7ef-c3c1-420e-91ed-79922eb9cb65-kube-api-access-47s9w\") pod \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\" (UID: \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\") "
Jul 20 10:21:27.392090 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.391815    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c559d7ef-c3c1-420e-91ed-79922eb9cb65-kserve-provision-location\") pod \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\" (UID: \"c559d7ef-c3c1-420e-91ed-79922eb9cb65\") "
Jul 20 10:21:27.392090 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.392055    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c559d7ef-c3c1-420e-91ed-79922eb9cb65-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "c559d7ef-c3c1-420e-91ed-79922eb9cb65" (UID: "c559d7ef-c3c1-420e-91ed-79922eb9cb65"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:21:27.392572 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.392548    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c559d7ef-c3c1-420e-91ed-79922eb9cb65-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c559d7ef-c3c1-420e-91ed-79922eb9cb65" (UID: "c559d7ef-c3c1-420e-91ed-79922eb9cb65"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:21:27.394146 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.394119    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c559d7ef-c3c1-420e-91ed-79922eb9cb65-kube-api-access-47s9w" (OuterVolumeSpecName: "kube-api-access-47s9w") pod "c559d7ef-c3c1-420e-91ed-79922eb9cb65" (UID: "c559d7ef-c3c1-420e-91ed-79922eb9cb65"). InnerVolumeSpecName "kube-api-access-47s9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:21:27.394146 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.394135    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c559d7ef-c3c1-420e-91ed-79922eb9cb65-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c559d7ef-c3c1-420e-91ed-79922eb9cb65" (UID: "c559d7ef-c3c1-420e-91ed-79922eb9cb65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:21:27.431542 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.431515    2580 generic.go:356] "Generic (PLEG): container finished" podID="c559d7ef-c3c1-420e-91ed-79922eb9cb65" containerID="db30d9da83652aad44d543a92ba4e6d9282faa8c59d45116a6de8de1689e6d54" exitCode=0
Jul 20 10:21:27.431682 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.431556    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"
Jul 20 10:21:27.431682 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.431571    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh" event={"ID":"c559d7ef-c3c1-420e-91ed-79922eb9cb65","Type":"ContainerDied","Data":"db30d9da83652aad44d543a92ba4e6d9282faa8c59d45116a6de8de1689e6d54"}
Jul 20 10:21:27.431682 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.431606    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh" event={"ID":"c559d7ef-c3c1-420e-91ed-79922eb9cb65","Type":"ContainerDied","Data":"1bf455ac974a606d970f8a668ffa6955a982fd4e50afc962cfb08706e83b25da"}
Jul 20 10:21:27.431682 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.431630    2580 scope.go:117] "RemoveContainer" containerID="db30d9da83652aad44d543a92ba4e6d9282faa8c59d45116a6de8de1689e6d54"
Jul 20 10:21:27.441156 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.441137    2580 scope.go:117] "RemoveContainer" containerID="4e9a3cd4ef5a9d5985dda17ef8845e2b380ffe1b292899b3aad1ee09b7b5adc0"
Jul 20 10:21:27.448945 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.448927    2580 scope.go:117] "RemoveContainer" containerID="db30d9da83652aad44d543a92ba4e6d9282faa8c59d45116a6de8de1689e6d54"
Jul 20 10:21:27.449219 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:21:27.449196    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db30d9da83652aad44d543a92ba4e6d9282faa8c59d45116a6de8de1689e6d54\": container with ID starting with db30d9da83652aad44d543a92ba4e6d9282faa8c59d45116a6de8de1689e6d54 not found: ID does not exist" containerID="db30d9da83652aad44d543a92ba4e6d9282faa8c59d45116a6de8de1689e6d54"
Jul 20 10:21:27.449324 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.449226    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db30d9da83652aad44d543a92ba4e6d9282faa8c59d45116a6de8de1689e6d54"} err="failed to get container status \"db30d9da83652aad44d543a92ba4e6d9282faa8c59d45116a6de8de1689e6d54\": rpc error: code = NotFound desc = could not find container \"db30d9da83652aad44d543a92ba4e6d9282faa8c59d45116a6de8de1689e6d54\": container with ID starting with db30d9da83652aad44d543a92ba4e6d9282faa8c59d45116a6de8de1689e6d54 not found: ID does not exist"
Jul 20 10:21:27.449324 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.449243    2580 scope.go:117] "RemoveContainer" containerID="4e9a3cd4ef5a9d5985dda17ef8845e2b380ffe1b292899b3aad1ee09b7b5adc0"
Jul 20 10:21:27.449505 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:21:27.449487    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e9a3cd4ef5a9d5985dda17ef8845e2b380ffe1b292899b3aad1ee09b7b5adc0\": container with ID starting with 4e9a3cd4ef5a9d5985dda17ef8845e2b380ffe1b292899b3aad1ee09b7b5adc0 not found: ID does not exist" containerID="4e9a3cd4ef5a9d5985dda17ef8845e2b380ffe1b292899b3aad1ee09b7b5adc0"
Jul 20 10:21:27.449541 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.449512    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e9a3cd4ef5a9d5985dda17ef8845e2b380ffe1b292899b3aad1ee09b7b5adc0"} err="failed to get container status \"4e9a3cd4ef5a9d5985dda17ef8845e2b380ffe1b292899b3aad1ee09b7b5adc0\": rpc error: code = NotFound desc = could not find container \"4e9a3cd4ef5a9d5985dda17ef8845e2b380ffe1b292899b3aad1ee09b7b5adc0\": container with ID starting with 4e9a3cd4ef5a9d5985dda17ef8845e2b380ffe1b292899b3aad1ee09b7b5adc0 not found: ID does not exist"
Jul 20 10:21:27.453666 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.453643    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"]
Jul 20 10:21:27.457331 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.457310    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-ddcb7bc45-dnllh"]
Jul 20 10:21:27.493553 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.493518    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c559d7ef-c3c1-420e-91ed-79922eb9cb65-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:21:27.493553 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.493550    2580 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c559d7ef-c3c1-420e-91ed-79922eb9cb65-tokenizer-uds\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:21:27.493805 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.493567    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47s9w\" (UniqueName: \"kubernetes.io/projected/c559d7ef-c3c1-420e-91ed-79922eb9cb65-kube-api-access-47s9w\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:21:27.493805 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:27.493583    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c559d7ef-c3c1-420e-91ed-79922eb9cb65-kserve-provision-location\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:21:28.411852 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:28.411822    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c559d7ef-c3c1-420e-91ed-79922eb9cb65" path="/var/lib/kubelet/pods/c559d7ef-c3c1-420e-91ed-79922eb9cb65/volumes"
Jul 20 10:21:33.315224 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:33.315155    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:33.400875 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:33.400839    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:40.146338 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:40.146302    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:44.222039 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.222001    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"]
Jul 20 10:21:44.222438 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.222342    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c559d7ef-c3c1-420e-91ed-79922eb9cb65" containerName="main"
Jul 20 10:21:44.222438 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.222354    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c559d7ef-c3c1-420e-91ed-79922eb9cb65" containerName="main"
Jul 20 10:21:44.222438 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.222364    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c559d7ef-c3c1-420e-91ed-79922eb9cb65" containerName="storage-initializer"
Jul 20 10:21:44.222438 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.222370    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c559d7ef-c3c1-420e-91ed-79922eb9cb65" containerName="storage-initializer"
Jul 20 10:21:44.222438 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.222390    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945d1d32-4f7d-4da2-9304-2413fc86b0dd" containerName="main"
Jul 20 10:21:44.222438 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.222395    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="945d1d32-4f7d-4da2-9304-2413fc86b0dd" containerName="main"
Jul 20 10:21:44.222438 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.222439    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="945d1d32-4f7d-4da2-9304-2413fc86b0dd" containerName="main"
Jul 20 10:21:44.222659 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.222448    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c559d7ef-c3c1-420e-91ed-79922eb9cb65" containerName="main"
Jul 20 10:21:44.226664 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.226647    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.229845 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.229820    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 20 10:21:44.230015 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.229863    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-hlwqc\""
Jul 20 10:21:44.230015 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.229873    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 20 10:21:44.230015 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.229936    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 20 10:21:44.239215 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.239192    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"]
Jul 20 10:21:44.258441 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.258387    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:44.327770 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.327732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.327770 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.327776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab46e8a-35b9-4854-9eae-ec967e435272-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.328025 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.327845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.328025 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.327901    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.328025 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.327926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk8xv\" (UniqueName: \"kubernetes.io/projected/aab46e8a-35b9-4854-9eae-ec967e435272-kube-api-access-nk8xv\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.328025 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.327971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.428505 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.428475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.428709 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.428515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab46e8a-35b9-4854-9eae-ec967e435272-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.428709 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.428540    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.428709 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.428585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.428709 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.428609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk8xv\" (UniqueName: \"kubernetes.io/projected/aab46e8a-35b9-4854-9eae-ec967e435272-kube-api-access-nk8xv\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.428997 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.428865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.428997 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.428922    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.428997 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.428976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.429113 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.429094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.431104 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.431088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.431190 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.431155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab46e8a-35b9-4854-9eae-ec967e435272-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.438724 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.438697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk8xv\" (UniqueName: \"kubernetes.io/projected/aab46e8a-35b9-4854-9eae-ec967e435272-kube-api-access-nk8xv\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.537463 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.537370    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:44.696529 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.696498    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"]
Jul 20 10:21:44.698378 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:21:44.698354    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab46e8a_35b9_4854_9eae_ec967e435272.slice/crio-c7c858734b5f09610cb7a11d6dbaf08981f003ce37736c1fa4ea80a71a75ea29 WatchSource:0}: Error finding container c7c858734b5f09610cb7a11d6dbaf08981f003ce37736c1fa4ea80a71a75ea29: Status 404 returned error can't find the container with id c7c858734b5f09610cb7a11d6dbaf08981f003ce37736c1fa4ea80a71a75ea29
Jul 20 10:21:44.710856 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:44.710831    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:45.528882 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:45.528846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4" event={"ID":"aab46e8a-35b9-4854-9eae-ec967e435272","Type":"ContainerStarted","Data":"89331283e9ee43bc7870410a2332624bfaad0b7e719c4b5c6b586a4e0612338b"}
Jul 20 10:21:45.529317 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:45.528925    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4" event={"ID":"aab46e8a-35b9-4854-9eae-ec967e435272","Type":"ContainerStarted","Data":"c7c858734b5f09610cb7a11d6dbaf08981f003ce37736c1fa4ea80a71a75ea29"}
Jul 20 10:21:45.537441 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:45.537420    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:45.546609 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:45.546565    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4" podStartSLOduration=1.5465247739999999 podStartE2EDuration="1.546524774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:21:45.545004022 +0000 UTC m=+1517.593867670" watchObservedRunningTime="2026-07-20 10:21:45.546524774 +0000 UTC m=+1517.595388420"
Jul 20 10:21:46.376607 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:46.376549    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:46.471861 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:46.471824    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:50.557104 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.557016    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"]
Jul 20 10:21:50.560440 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.560425    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"
Jul 20 10:21:50.562612 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.562587    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-m44gb\""
Jul 20 10:21:50.562744 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.562656    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-np6z8\""
Jul 20 10:21:50.563127 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.563107    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Jul 20 10:21:50.563281 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.563260    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Jul 20 10:21:50.563281 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.563274    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 20 10:21:50.563410 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.563278    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Jul 20 10:21:50.572558 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.572528    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"]
Jul 20 10:21:50.575569 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.575546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb8j9\" (UniqueName: \"kubernetes.io/projected/1470a265-be01-4b34-b8cb-32d8fa309606-kube-api-access-sb8j9\") pod \"auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp\" (UID: \"1470a265-be01-4b34-b8cb-32d8fa309606\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"
Jul 20 10:21:50.575679 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.575594    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1470a265-be01-4b34-b8cb-32d8fa309606-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp\" (UID: \"1470a265-be01-4b34-b8cb-32d8fa309606\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"
Jul 20 10:21:50.575745 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.575677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1470a265-be01-4b34-b8cb-32d8fa309606-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp\" (UID: \"1470a265-be01-4b34-b8cb-32d8fa309606\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"
Jul 20 10:21:50.575745 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.575720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1470a265-be01-4b34-b8cb-32d8fa309606-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp\" (UID: \"1470a265-be01-4b34-b8cb-32d8fa309606\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"
Jul 20 10:21:50.676426 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.676392    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1470a265-be01-4b34-b8cb-32d8fa309606-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp\" (UID: \"1470a265-be01-4b34-b8cb-32d8fa309606\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"
Jul 20 10:21:50.676628 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.676434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1470a265-be01-4b34-b8cb-32d8fa309606-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp\" (UID: \"1470a265-be01-4b34-b8cb-32d8fa309606\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"
Jul 20 10:21:50.676628 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.676472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb8j9\" (UniqueName: \"kubernetes.io/projected/1470a265-be01-4b34-b8cb-32d8fa309606-kube-api-access-sb8j9\") pod \"auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp\" (UID: \"1470a265-be01-4b34-b8cb-32d8fa309606\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"
Jul 20 10:21:50.676628 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.676498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1470a265-be01-4b34-b8cb-32d8fa309606-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp\" (UID: \"1470a265-be01-4b34-b8cb-32d8fa309606\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"
Jul 20 10:21:50.676806 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.676774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1470a265-be01-4b34-b8cb-32d8fa309606-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp\" (UID: \"1470a265-be01-4b34-b8cb-32d8fa309606\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"
Jul 20 10:21:50.676907 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.676886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1470a265-be01-4b34-b8cb-32d8fa309606-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp\" (UID: \"1470a265-be01-4b34-b8cb-32d8fa309606\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"
Jul 20 10:21:50.678990 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.678968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1470a265-be01-4b34-b8cb-32d8fa309606-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp\" (UID: \"1470a265-be01-4b34-b8cb-32d8fa309606\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"
Jul 20 10:21:50.692617 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.692577    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb8j9\" (UniqueName: \"kubernetes.io/projected/1470a265-be01-4b34-b8cb-32d8fa309606-kube-api-access-sb8j9\") pod \"auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp\" (UID: \"1470a265-be01-4b34-b8cb-32d8fa309606\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"
Jul 20 10:21:50.874384 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:50.874280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"
Jul 20 10:21:51.019690 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:51.019653    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"]
Jul 20 10:21:51.023051 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:21:51.023018    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1470a265_be01_4b34_b8cb_32d8fa309606.slice/crio-00d9833ab41412b3d0f9bb7cbf5013ba5fd3bab0b60292b973e540fa26938c2e WatchSource:0}: Error finding container 00d9833ab41412b3d0f9bb7cbf5013ba5fd3bab0b60292b973e540fa26938c2e: Status 404 returned error can't find the container with id 00d9833ab41412b3d0f9bb7cbf5013ba5fd3bab0b60292b973e540fa26938c2e
Jul 20 10:21:51.040486 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:51.040352    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:51.145562 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:51.145458    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:51.564293 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:51.564256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp" event={"ID":"1470a265-be01-4b34-b8cb-32d8fa309606","Type":"ContainerStarted","Data":"59165da3d151187dd737d144f6935ef9e6f0ed79810e59dd0b1b3bfd3bba36e2"}
Jul 20 10:21:51.564721 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:51.564327    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp" event={"ID":"1470a265-be01-4b34-b8cb-32d8fa309606","Type":"ContainerStarted","Data":"00d9833ab41412b3d0f9bb7cbf5013ba5fd3bab0b60292b973e540fa26938c2e"}
Jul 20 10:21:52.570687 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:52.570651    2580 generic.go:356] "Generic (PLEG): container finished" podID="1470a265-be01-4b34-b8cb-32d8fa309606" containerID="59165da3d151187dd737d144f6935ef9e6f0ed79810e59dd0b1b3bfd3bba36e2" exitCode=0
Jul 20 10:21:52.571159 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:52.570733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp" event={"ID":"1470a265-be01-4b34-b8cb-32d8fa309606","Type":"ContainerDied","Data":"59165da3d151187dd737d144f6935ef9e6f0ed79810e59dd0b1b3bfd3bba36e2"}
Jul 20 10:21:53.578034 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:53.578003    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp" event={"ID":"1470a265-be01-4b34-b8cb-32d8fa309606","Type":"ContainerStarted","Data":"cdc7e5980853996afb30d31a3e6e73fda77e7e7a12ae204130e1250090840981"}
Jul 20 10:21:53.578464 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:53.578127    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"
Jul 20 10:21:53.601435 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:53.601382    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp" podStartSLOduration=2.951931866 podStartE2EDuration="3.601368105s" totalImagesPullingTime="0s" totalInitContainerRuntime="649.436239ms" isStatefulPod=false podCreationTimestamp="2026-07-20 10:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:21:53.598852333 +0000 UTC m=+1525.647715981" watchObservedRunningTime="2026-07-20 10:21:53.601368105 +0000 UTC m=+1525.650231751"
Jul 20 10:21:54.537711 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:54.537662    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:54.557869 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:54.557839    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:21:54.594049 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:21:54.594021    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:22:01.691055 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:01.691022    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:22:13.399428 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:13.399390    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:22:20.363287 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:20.363241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:22:23.918445 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:23.918412    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"]
Jul 20 10:22:23.920580 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:23.920537    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:22:23.928928 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:22:23.928897    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:22:23.928928 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:22:23.928928 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:22:23.928928 ip-10-0-133-160 kubenswrapper[2580]:  > containerID="89331283e9ee43bc7870410a2332624bfaad0b7e719c4b5c6b586a4e0612338b" cmd=["/bin/sleep","15"]
Jul 20 10:22:23.929207 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:23.928944    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4" message="" err=<
Jul 20 10:22:23.929207 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:22:23.929207 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:22:23.929207 ip-10-0-133-160 kubenswrapper[2580]:  >
Jul 20 10:22:23.929207 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:22:23.928955    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:22:23.929207 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:22:23.929207 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 10:22:23.929207 ip-10-0-133-160 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4" podUID="aab46e8a-35b9-4854-9eae-ec967e435272" containerName="main" containerID="cri-o://89331283e9ee43bc7870410a2332624bfaad0b7e719c4b5c6b586a4e0612338b"
Jul 20 10:22:23.929207 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:23.929006    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4" podUID="aab46e8a-35b9-4854-9eae-ec967e435272" containerName="main" containerID="cri-o://89331283e9ee43bc7870410a2332624bfaad0b7e719c4b5c6b586a4e0612338b" gracePeriod=60
Jul 20 10:22:24.018482 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.018443    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:22:24.286026 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.286001    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:22:24.387804 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.387766    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-home\") pod \"aab46e8a-35b9-4854-9eae-ec967e435272\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") "
Jul 20 10:22:24.388008 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.387808    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab46e8a-35b9-4854-9eae-ec967e435272-tls-certs\") pod \"aab46e8a-35b9-4854-9eae-ec967e435272\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") "
Jul 20 10:22:24.388008 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.387912    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-model-cache\") pod \"aab46e8a-35b9-4854-9eae-ec967e435272\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") "
Jul 20 10:22:24.388008 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.387971    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nk8xv\" (UniqueName: \"kubernetes.io/projected/aab46e8a-35b9-4854-9eae-ec967e435272-kube-api-access-nk8xv\") pod \"aab46e8a-35b9-4854-9eae-ec967e435272\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") "
Jul 20 10:22:24.388227 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.388029    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-tmp-dir\") pod \"aab46e8a-35b9-4854-9eae-ec967e435272\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") "
Jul 20 10:22:24.388227 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.388095    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-dshm\") pod \"aab46e8a-35b9-4854-9eae-ec967e435272\" (UID: \"aab46e8a-35b9-4854-9eae-ec967e435272\") "
Jul 20 10:22:24.388227 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.388100    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-home" (OuterVolumeSpecName: "home") pod "aab46e8a-35b9-4854-9eae-ec967e435272" (UID: "aab46e8a-35b9-4854-9eae-ec967e435272"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:22:24.388227 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.388138    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-model-cache" (OuterVolumeSpecName: "model-cache") pod "aab46e8a-35b9-4854-9eae-ec967e435272" (UID: "aab46e8a-35b9-4854-9eae-ec967e435272"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:22:24.388403 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.388326    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aab46e8a-35b9-4854-9eae-ec967e435272" (UID: "aab46e8a-35b9-4854-9eae-ec967e435272"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:22:24.388466 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.388446    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-tmp-dir\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:22:24.388526 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.388471    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-home\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:22:24.388526 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.388484    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-model-cache\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:22:24.390393 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.390362    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-dshm" (OuterVolumeSpecName: "dshm") pod "aab46e8a-35b9-4854-9eae-ec967e435272" (UID: "aab46e8a-35b9-4854-9eae-ec967e435272"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:22:24.390505 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.390417    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab46e8a-35b9-4854-9eae-ec967e435272-kube-api-access-nk8xv" (OuterVolumeSpecName: "kube-api-access-nk8xv") pod "aab46e8a-35b9-4854-9eae-ec967e435272" (UID: "aab46e8a-35b9-4854-9eae-ec967e435272"). InnerVolumeSpecName "kube-api-access-nk8xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:22:24.390505 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.390438    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aab46e8a-35b9-4854-9eae-ec967e435272-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aab46e8a-35b9-4854-9eae-ec967e435272" (UID: "aab46e8a-35b9-4854-9eae-ec967e435272"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:22:24.489057 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.489027    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab46e8a-35b9-4854-9eae-ec967e435272-dshm\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:22:24.489057 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.489055    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab46e8a-35b9-4854-9eae-ec967e435272-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:22:24.489298 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.489066    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk8xv\" (UniqueName: \"kubernetes.io/projected/aab46e8a-35b9-4854-9eae-ec967e435272-kube-api-access-nk8xv\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:22:24.585392 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.585365    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"
Jul 20 10:22:24.735944 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.735902    2580 generic.go:356] "Generic (PLEG): container finished" podID="aab46e8a-35b9-4854-9eae-ec967e435272" containerID="89331283e9ee43bc7870410a2332624bfaad0b7e719c4b5c6b586a4e0612338b" exitCode=0
Jul 20 10:22:24.736139 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.735957    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"
Jul 20 10:22:24.736139 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.735978    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4" event={"ID":"aab46e8a-35b9-4854-9eae-ec967e435272","Type":"ContainerDied","Data":"89331283e9ee43bc7870410a2332624bfaad0b7e719c4b5c6b586a4e0612338b"}
Jul 20 10:22:24.736139 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.736037    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4" event={"ID":"aab46e8a-35b9-4854-9eae-ec967e435272","Type":"ContainerDied","Data":"c7c858734b5f09610cb7a11d6dbaf08981f003ce37736c1fa4ea80a71a75ea29"}
Jul 20 10:22:24.736139 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.736068    2580 scope.go:117] "RemoveContainer" containerID="89331283e9ee43bc7870410a2332624bfaad0b7e719c4b5c6b586a4e0612338b"
Jul 20 10:22:24.744935 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.744913    2580 scope.go:117] "RemoveContainer" containerID="89331283e9ee43bc7870410a2332624bfaad0b7e719c4b5c6b586a4e0612338b"
Jul 20 10:22:24.745206 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:22:24.745190    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89331283e9ee43bc7870410a2332624bfaad0b7e719c4b5c6b586a4e0612338b\": container with ID starting with 89331283e9ee43bc7870410a2332624bfaad0b7e719c4b5c6b586a4e0612338b not found: ID does not exist" containerID="89331283e9ee43bc7870410a2332624bfaad0b7e719c4b5c6b586a4e0612338b"
Jul 20 10:22:24.745259 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.745213    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89331283e9ee43bc7870410a2332624bfaad0b7e719c4b5c6b586a4e0612338b"} err="failed to get container status \"89331283e9ee43bc7870410a2332624bfaad0b7e719c4b5c6b586a4e0612338b\": rpc error: code = NotFound desc = could not find container \"89331283e9ee43bc7870410a2332624bfaad0b7e719c4b5c6b586a4e0612338b\": container with ID starting with 89331283e9ee43bc7870410a2332624bfaad0b7e719c4b5c6b586a4e0612338b not found: ID does not exist"
Jul 20 10:22:24.765364 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.765336    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"]
Jul 20 10:22:24.767723 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:24.767700    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7kvx4"]
Jul 20 10:22:26.411410 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:26.411376    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aab46e8a-35b9-4854-9eae-ec967e435272" path="/var/lib/kubelet/pods/aab46e8a-35b9-4854-9eae-ec967e435272/volumes"
Jul 20 10:22:43.783786 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:43.783748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:22:43.848305 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:22:43.848267    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:23:01.576114 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:23:01.576063    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:23:03.805557 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:23:03.805521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:23:03.902089 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:23:03.902050    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:23:10.340254 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:23:10.340219    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:23:13.545840 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:23:13.545803    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:23:13.855083 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:23:13.854921    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:23:16.528603 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:23:16.528565    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:23:16.627212 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:23:16.627161    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:23:30.604368 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:23:30.604331    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:23:55.618680 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:23:55.618647    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:23:55.632126 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:23:55.632096    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"]
Jul 20 10:23:55.752760 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:23:55.752724    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:24:04.585004 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:04.584964    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp" podUID="1470a265-be01-4b34-b8cb-32d8fa309606" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:24:10.652685 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:10.652639    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp" podUID="1470a265-be01-4b34-b8cb-32d8fa309606" containerName="main" containerID="cri-o://cdc7e5980853996afb30d31a3e6e73fda77e7e7a12ae204130e1250090840981" gracePeriod=45
Jul 20 10:24:10.928790 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:10.928764    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"
Jul 20 10:24:11.118491 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.118454    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1470a265-be01-4b34-b8cb-32d8fa309606-tls-certs\") pod \"1470a265-be01-4b34-b8cb-32d8fa309606\" (UID: \"1470a265-be01-4b34-b8cb-32d8fa309606\") "
Jul 20 10:24:11.118709 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.118508    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1470a265-be01-4b34-b8cb-32d8fa309606-kserve-provision-location\") pod \"1470a265-be01-4b34-b8cb-32d8fa309606\" (UID: \"1470a265-be01-4b34-b8cb-32d8fa309606\") "
Jul 20 10:24:11.118709 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.118561    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb8j9\" (UniqueName: \"kubernetes.io/projected/1470a265-be01-4b34-b8cb-32d8fa309606-kube-api-access-sb8j9\") pod \"1470a265-be01-4b34-b8cb-32d8fa309606\" (UID: \"1470a265-be01-4b34-b8cb-32d8fa309606\") "
Jul 20 10:24:11.118709 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.118582    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1470a265-be01-4b34-b8cb-32d8fa309606-tokenizer-uds\") pod \"1470a265-be01-4b34-b8cb-32d8fa309606\" (UID: \"1470a265-be01-4b34-b8cb-32d8fa309606\") "
Jul 20 10:24:11.118895 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.118866    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1470a265-be01-4b34-b8cb-32d8fa309606-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1470a265-be01-4b34-b8cb-32d8fa309606" (UID: "1470a265-be01-4b34-b8cb-32d8fa309606"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:24:11.119096 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.119080    2580 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1470a265-be01-4b34-b8cb-32d8fa309606-tokenizer-uds\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:24:11.119436 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.119348    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1470a265-be01-4b34-b8cb-32d8fa309606-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1470a265-be01-4b34-b8cb-32d8fa309606" (UID: "1470a265-be01-4b34-b8cb-32d8fa309606"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:24:11.121042 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.121018    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1470a265-be01-4b34-b8cb-32d8fa309606-kube-api-access-sb8j9" (OuterVolumeSpecName: "kube-api-access-sb8j9") pod "1470a265-be01-4b34-b8cb-32d8fa309606" (UID: "1470a265-be01-4b34-b8cb-32d8fa309606"). InnerVolumeSpecName "kube-api-access-sb8j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:24:11.121313 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.121289    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1470a265-be01-4b34-b8cb-32d8fa309606-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1470a265-be01-4b34-b8cb-32d8fa309606" (UID: "1470a265-be01-4b34-b8cb-32d8fa309606"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:24:11.220009 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.219979    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1470a265-be01-4b34-b8cb-32d8fa309606-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:24:11.220009 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.220005    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1470a265-be01-4b34-b8cb-32d8fa309606-kserve-provision-location\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:24:11.220240 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.220017    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb8j9\" (UniqueName: \"kubernetes.io/projected/1470a265-be01-4b34-b8cb-32d8fa309606-kube-api-access-sb8j9\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:24:11.310246 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.310210    2580 generic.go:356] "Generic (PLEG): container finished" podID="1470a265-be01-4b34-b8cb-32d8fa309606" containerID="cdc7e5980853996afb30d31a3e6e73fda77e7e7a12ae204130e1250090840981" exitCode=0
Jul 20 10:24:11.310448 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.310313    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"
Jul 20 10:24:11.310448 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.310312    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp" event={"ID":"1470a265-be01-4b34-b8cb-32d8fa309606","Type":"ContainerDied","Data":"cdc7e5980853996afb30d31a3e6e73fda77e7e7a12ae204130e1250090840981"}
Jul 20 10:24:11.310448 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.310433    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp" event={"ID":"1470a265-be01-4b34-b8cb-32d8fa309606","Type":"ContainerDied","Data":"00d9833ab41412b3d0f9bb7cbf5013ba5fd3bab0b60292b973e540fa26938c2e"}
Jul 20 10:24:11.310583 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.310460    2580 scope.go:117] "RemoveContainer" containerID="cdc7e5980853996afb30d31a3e6e73fda77e7e7a12ae204130e1250090840981"
Jul 20 10:24:11.321558 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.321535    2580 scope.go:117] "RemoveContainer" containerID="59165da3d151187dd737d144f6935ef9e6f0ed79810e59dd0b1b3bfd3bba36e2"
Jul 20 10:24:11.333213 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.333190    2580 scope.go:117] "RemoveContainer" containerID="cdc7e5980853996afb30d31a3e6e73fda77e7e7a12ae204130e1250090840981"
Jul 20 10:24:11.333967 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:24:11.333919    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdc7e5980853996afb30d31a3e6e73fda77e7e7a12ae204130e1250090840981\": container with ID starting with cdc7e5980853996afb30d31a3e6e73fda77e7e7a12ae204130e1250090840981 not found: ID does not exist" containerID="cdc7e5980853996afb30d31a3e6e73fda77e7e7a12ae204130e1250090840981"
Jul 20 10:24:11.334088 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.333966    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc7e5980853996afb30d31a3e6e73fda77e7e7a12ae204130e1250090840981"} err="failed to get container status \"cdc7e5980853996afb30d31a3e6e73fda77e7e7a12ae204130e1250090840981\": rpc error: code = NotFound desc = could not find container \"cdc7e5980853996afb30d31a3e6e73fda77e7e7a12ae204130e1250090840981\": container with ID starting with cdc7e5980853996afb30d31a3e6e73fda77e7e7a12ae204130e1250090840981 not found: ID does not exist"
Jul 20 10:24:11.334088 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.333994    2580 scope.go:117] "RemoveContainer" containerID="59165da3d151187dd737d144f6935ef9e6f0ed79810e59dd0b1b3bfd3bba36e2"
Jul 20 10:24:11.334435 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:24:11.334410    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59165da3d151187dd737d144f6935ef9e6f0ed79810e59dd0b1b3bfd3bba36e2\": container with ID starting with 59165da3d151187dd737d144f6935ef9e6f0ed79810e59dd0b1b3bfd3bba36e2 not found: ID does not exist" containerID="59165da3d151187dd737d144f6935ef9e6f0ed79810e59dd0b1b3bfd3bba36e2"
Jul 20 10:24:11.334531 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.334447    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59165da3d151187dd737d144f6935ef9e6f0ed79810e59dd0b1b3bfd3bba36e2"} err="failed to get container status \"59165da3d151187dd737d144f6935ef9e6f0ed79810e59dd0b1b3bfd3bba36e2\": rpc error: code = NotFound desc = could not find container \"59165da3d151187dd737d144f6935ef9e6f0ed79810e59dd0b1b3bfd3bba36e2\": container with ID starting with 59165da3d151187dd737d144f6935ef9e6f0ed79810e59dd0b1b3bfd3bba36e2 not found: ID does not exist"
Jul 20 10:24:11.335954 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.335937    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"]
Jul 20 10:24:11.339637 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:11.339613    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-67fc747c57-fqxxp"]
Jul 20 10:24:12.411544 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:12.411506    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1470a265-be01-4b34-b8cb-32d8fa309606" path="/var/lib/kubelet/pods/1470a265-be01-4b34-b8cb-32d8fa309606/volumes"
Jul 20 10:24:15.531727 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:15.531692    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:24:15.632142 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:15.632110    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:24:21.557398 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:21.557362    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:24:22.391243 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:22.391206    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:24:28.103774 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:28.103733    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:24:28.157135 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:28.157099    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:24:31.993390 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:31.993352    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:24:32.046658 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:24:32.046628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:25:09.359567 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:25:09.359463    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:25:09.421589 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:25:09.421550    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:25:29.045393 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:25:29.045347    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:25:29.139447 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:25:29.139415    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:25:35.359655 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:25:35.359619    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:25:40.909501 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:25:40.909462    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:25:40.978003 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:25:40.977969    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:25:46.558112 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:25:46.558073    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:25:46.791696 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:25:46.791646    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:26:00.808384 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:26:00.808349    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:26:47.697792 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:26:47.697757    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:27:08.732415 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:27:08.732373    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:27:08.798431 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:27:08.798395    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:27:28.722466 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:27:28.722432    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:27:28.821570 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:27:28.821530    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:27:28.918280 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:27:28.918250    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:27:29.018353 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:27:29.018241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:27:29.118609 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:27:29.118573    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:27:31.568685 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:27:31.568647    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:27:31.627344 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:27:31.627303    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:27:32.067507 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:27:32.067468    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:27:34.846494 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:27:34.846447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:27:40.549149 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:27:40.549110    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:27:40.608273 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:27:40.608236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:27:47.616827 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:27:47.616785    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:28:01.320603 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.320516    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:28:01.537515 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.537480    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"]
Jul 20 10:28:01.537833 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.537814    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aab46e8a-35b9-4854-9eae-ec967e435272" containerName="main"
Jul 20 10:28:01.537833 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.537831    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aab46e8a-35b9-4854-9eae-ec967e435272" containerName="main"
Jul 20 10:28:01.538029 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.537853    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1470a265-be01-4b34-b8cb-32d8fa309606" containerName="storage-initializer"
Jul 20 10:28:01.538029 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.537864    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1470a265-be01-4b34-b8cb-32d8fa309606" containerName="storage-initializer"
Jul 20 10:28:01.538029 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.537884    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1470a265-be01-4b34-b8cb-32d8fa309606" containerName="main"
Jul 20 10:28:01.538029 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.537892    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1470a265-be01-4b34-b8cb-32d8fa309606" containerName="main"
Jul 20 10:28:01.538029 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.537970    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1470a265-be01-4b34-b8cb-32d8fa309606" containerName="main"
Jul 20 10:28:01.538029 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.537988    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aab46e8a-35b9-4854-9eae-ec967e435272" containerName="main"
Jul 20 10:28:01.541119 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.541099    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"
Jul 20 10:28:01.545620 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.545455    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 20 10:28:01.545620 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.545455    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-5fbz5\""
Jul 20 10:28:01.545620 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.545507    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-p6sgc\""
Jul 20 10:28:01.545620 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.545546    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 20 10:28:01.545620 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.545509    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 20 10:28:01.556312 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.556283    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"]
Jul 20 10:28:01.588739 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.588633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gllv5\" (UniqueName: \"kubernetes.io/projected/7f246f26-5da0-4f10-bb45-4821ae9b9e4e-kube-api-access-gllv5\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb\" (UID: \"7f246f26-5da0-4f10-bb45-4821ae9b9e4e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"
Jul 20 10:28:01.588739 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.588716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7f246f26-5da0-4f10-bb45-4821ae9b9e4e-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb\" (UID: \"7f246f26-5da0-4f10-bb45-4821ae9b9e4e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"
Jul 20 10:28:01.588972 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.588769    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f246f26-5da0-4f10-bb45-4821ae9b9e4e-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb\" (UID: \"7f246f26-5da0-4f10-bb45-4821ae9b9e4e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"
Jul 20 10:28:01.593034 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.592995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:28:01.690105 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.690059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gllv5\" (UniqueName: \"kubernetes.io/projected/7f246f26-5da0-4f10-bb45-4821ae9b9e4e-kube-api-access-gllv5\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb\" (UID: \"7f246f26-5da0-4f10-bb45-4821ae9b9e4e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"
Jul 20 10:28:01.690331 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.690129    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7f246f26-5da0-4f10-bb45-4821ae9b9e4e-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb\" (UID: \"7f246f26-5da0-4f10-bb45-4821ae9b9e4e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"
Jul 20 10:28:01.690331 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.690213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f246f26-5da0-4f10-bb45-4821ae9b9e4e-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb\" (UID: \"7f246f26-5da0-4f10-bb45-4821ae9b9e4e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"
Jul 20 10:28:01.690552 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.690533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7f246f26-5da0-4f10-bb45-4821ae9b9e4e-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb\" (UID: \"7f246f26-5da0-4f10-bb45-4821ae9b9e4e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"
Jul 20 10:28:01.692806 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.692784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f246f26-5da0-4f10-bb45-4821ae9b9e4e-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb\" (UID: \"7f246f26-5da0-4f10-bb45-4821ae9b9e4e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"
Jul 20 10:28:01.699627 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.699600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gllv5\" (UniqueName: \"kubernetes.io/projected/7f246f26-5da0-4f10-bb45-4821ae9b9e4e-kube-api-access-gllv5\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb\" (UID: \"7f246f26-5da0-4f10-bb45-4821ae9b9e4e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"
Jul 20 10:28:01.855534 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.855441    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"
Jul 20 10:28:01.995444 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:01.995419    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"]
Jul 20 10:28:01.997826 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:28:01.997799    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f246f26_5da0_4f10_bb45_4821ae9b9e4e.slice/crio-03f9e5fa4e018796ed9da96dd22e1b9afbc63f2824b63543312351cc74fc4a4d WatchSource:0}: Error finding container 03f9e5fa4e018796ed9da96dd22e1b9afbc63f2824b63543312351cc74fc4a4d: Status 404 returned error can't find the container with id 03f9e5fa4e018796ed9da96dd22e1b9afbc63f2824b63543312351cc74fc4a4d
Jul 20 10:28:02.463687 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:02.463654    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb" event={"ID":"7f246f26-5da0-4f10-bb45-4821ae9b9e4e","Type":"ContainerStarted","Data":"7c451697fc1cb4539402ed39542db00e1f6159de771230853e69cecc78ca5174"}
Jul 20 10:28:02.464155 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:02.463713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb" event={"ID":"7f246f26-5da0-4f10-bb45-4821ae9b9e4e","Type":"ContainerStarted","Data":"03f9e5fa4e018796ed9da96dd22e1b9afbc63f2824b63543312351cc74fc4a4d"}
Jul 20 10:28:02.464155 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:02.463866    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"
Jul 20 10:28:02.493797 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:02.493745    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb" podStartSLOduration=1.493728869 podStartE2EDuration="1.493728869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 10:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:28:02.492470161 +0000 UTC m=+1894.541333847" watchObservedRunningTime="2026-07-20 10:28:02.493728869 +0000 UTC m=+1894.542592548"
Jul 20 10:28:11.486195 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:11.486136    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:28:19.682746 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:19.682707    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:28:32.829101 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:32.829068    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:28:33.470867 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:28:33.470840    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"
Jul 20 10:30:14.000692 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:14.000653    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"]
Jul 20 10:30:14.037417 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:14.037381    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:14.098750 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:14.098706    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:23.471315 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:23.471279    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb" podUID="7f246f26-5da0-4f10-bb45-4821ae9b9e4e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:30:29.024438 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:29.024373    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb" podUID="7f246f26-5da0-4f10-bb45-4821ae9b9e4e" containerName="main" containerID="cri-o://7c451697fc1cb4539402ed39542db00e1f6159de771230853e69cecc78ca5174" gracePeriod=45
Jul 20 10:30:29.201704 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:29.201672    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f246f26-5da0-4f10-bb45-4821ae9b9e4e" containerID="7c451697fc1cb4539402ed39542db00e1f6159de771230853e69cecc78ca5174" exitCode=0
Jul 20 10:30:29.201836 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:29.201739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb" event={"ID":"7f246f26-5da0-4f10-bb45-4821ae9b9e4e","Type":"ContainerDied","Data":"7c451697fc1cb4539402ed39542db00e1f6159de771230853e69cecc78ca5174"}
Jul 20 10:30:29.284648 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:29.284571    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"
Jul 20 10:30:29.411444 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:29.411406    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f246f26-5da0-4f10-bb45-4821ae9b9e4e-tls-certs\") pod \"7f246f26-5da0-4f10-bb45-4821ae9b9e4e\" (UID: \"7f246f26-5da0-4f10-bb45-4821ae9b9e4e\") "
Jul 20 10:30:29.411625 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:29.411471    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7f246f26-5da0-4f10-bb45-4821ae9b9e4e-tokenizer-uds\") pod \"7f246f26-5da0-4f10-bb45-4821ae9b9e4e\" (UID: \"7f246f26-5da0-4f10-bb45-4821ae9b9e4e\") "
Jul 20 10:30:29.411625 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:29.411504    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gllv5\" (UniqueName: \"kubernetes.io/projected/7f246f26-5da0-4f10-bb45-4821ae9b9e4e-kube-api-access-gllv5\") pod \"7f246f26-5da0-4f10-bb45-4821ae9b9e4e\" (UID: \"7f246f26-5da0-4f10-bb45-4821ae9b9e4e\") "
Jul 20 10:30:29.411749 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:29.411725    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f246f26-5da0-4f10-bb45-4821ae9b9e4e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "7f246f26-5da0-4f10-bb45-4821ae9b9e4e" (UID: "7f246f26-5da0-4f10-bb45-4821ae9b9e4e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:30:29.411808 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:29.411771    2580 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7f246f26-5da0-4f10-bb45-4821ae9b9e4e-tokenizer-uds\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:30:29.413856 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:29.413833    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f246f26-5da0-4f10-bb45-4821ae9b9e4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f246f26-5da0-4f10-bb45-4821ae9b9e4e" (UID: "7f246f26-5da0-4f10-bb45-4821ae9b9e4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:30:29.413941 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:29.413845    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f246f26-5da0-4f10-bb45-4821ae9b9e4e-kube-api-access-gllv5" (OuterVolumeSpecName: "kube-api-access-gllv5") pod "7f246f26-5da0-4f10-bb45-4821ae9b9e4e" (UID: "7f246f26-5da0-4f10-bb45-4821ae9b9e4e"). InnerVolumeSpecName "kube-api-access-gllv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:30:29.513127 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:29.513091    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f246f26-5da0-4f10-bb45-4821ae9b9e4e-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:30:29.513127 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:29.513120    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gllv5\" (UniqueName: \"kubernetes.io/projected/7f246f26-5da0-4f10-bb45-4821ae9b9e4e-kube-api-access-gllv5\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:30:30.208434 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:30.208402    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"
Jul 20 10:30:30.208882 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:30.208443    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb" event={"ID":"7f246f26-5da0-4f10-bb45-4821ae9b9e4e","Type":"ContainerDied","Data":"03f9e5fa4e018796ed9da96dd22e1b9afbc63f2824b63543312351cc74fc4a4d"}
Jul 20 10:30:30.208882 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:30.208507    2580 scope.go:117] "RemoveContainer" containerID="7c451697fc1cb4539402ed39542db00e1f6159de771230853e69cecc78ca5174"
Jul 20 10:30:30.229493 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:30.229465    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"]
Jul 20 10:30:30.233239 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:30.233213    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schernzxb"]
Jul 20 10:30:30.411852 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:30.411822    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f246f26-5da0-4f10-bb45-4821ae9b9e4e" path="/var/lib/kubelet/pods/7f246f26-5da0-4f10-bb45-4821ae9b9e4e/volumes"
Jul 20 10:30:34.004653 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:34.004615    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:34.100409 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:34.100368    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:34.202503 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:34.202449    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:34.309454 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:34.309359    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:34.409072 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:34.409008    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:36.681075 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:36.681036    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:36.742588 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:36.742556    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:37.483783 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:37.483738    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:39.237378 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:39.237335    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:39.304087 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:39.304049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:39.480014 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:39.479974    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:39.593652 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:39.593548    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:40.022709 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:40.022669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:45.871109 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:45.871071    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:45.971650 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:30:45.971615    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:31:03.556157 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.556063    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:31:03.752606 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.752562    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"]
Jul 20 10:31:03.753135 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.753114    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f246f26-5da0-4f10-bb45-4821ae9b9e4e" containerName="main"
Jul 20 10:31:03.753193 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.753141    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f246f26-5da0-4f10-bb45-4821ae9b9e4e" containerName="main"
Jul 20 10:31:03.753299 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.753286    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f246f26-5da0-4f10-bb45-4821ae9b9e4e" containerName="main"
Jul 20 10:31:03.756723 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.756699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"
Jul 20 10:31:03.760285 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.760253    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 20 10:31:03.760745 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.760722    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-kvcql\""
Jul 20 10:31:03.761000 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.760828    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 20 10:31:03.762317 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.762284    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 20 10:31:03.762433 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.762324    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-8455f\""
Jul 20 10:31:03.773733 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.773708    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"]
Jul 20 10:31:03.809709 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.809616    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/597e2c52-5f4f-43de-81c7-f530332222cf-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z\" (UID: \"597e2c52-5f4f-43de-81c7-f530332222cf\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"
Jul 20 10:31:03.809856 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.809726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rszzn\" (UniqueName: \"kubernetes.io/projected/597e2c52-5f4f-43de-81c7-f530332222cf-kube-api-access-rszzn\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z\" (UID: \"597e2c52-5f4f-43de-81c7-f530332222cf\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"
Jul 20 10:31:03.809856 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.809756    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/597e2c52-5f4f-43de-81c7-f530332222cf-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z\" (UID: \"597e2c52-5f4f-43de-81c7-f530332222cf\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"
Jul 20 10:31:03.845439 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.845405    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:31:03.910425 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.910386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rszzn\" (UniqueName: \"kubernetes.io/projected/597e2c52-5f4f-43de-81c7-f530332222cf-kube-api-access-rszzn\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z\" (UID: \"597e2c52-5f4f-43de-81c7-f530332222cf\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"
Jul 20 10:31:03.910425 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.910427    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/597e2c52-5f4f-43de-81c7-f530332222cf-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z\" (UID: \"597e2c52-5f4f-43de-81c7-f530332222cf\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"
Jul 20 10:31:03.910696 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.910492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/597e2c52-5f4f-43de-81c7-f530332222cf-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z\" (UID: \"597e2c52-5f4f-43de-81c7-f530332222cf\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"
Jul 20 10:31:03.910914 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.910893    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/597e2c52-5f4f-43de-81c7-f530332222cf-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z\" (UID: \"597e2c52-5f4f-43de-81c7-f530332222cf\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"
Jul 20 10:31:03.913066 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.913048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/597e2c52-5f4f-43de-81c7-f530332222cf-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z\" (UID: \"597e2c52-5f4f-43de-81c7-f530332222cf\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"
Jul 20 10:31:03.918036 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:03.918011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rszzn\" (UniqueName: \"kubernetes.io/projected/597e2c52-5f4f-43de-81c7-f530332222cf-kube-api-access-rszzn\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z\" (UID: \"597e2c52-5f4f-43de-81c7-f530332222cf\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"
Jul 20 10:31:04.069341 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:04.069243    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"
Jul 20 10:31:04.207976 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:04.207939    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"]
Jul 20 10:31:04.383950 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:04.383860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z" event={"ID":"597e2c52-5f4f-43de-81c7-f530332222cf","Type":"ContainerStarted","Data":"b833abb2de2b3526a1164f5d4995444c2fc4514d34af23d95558fb9ddcee0c53"}
Jul 20 10:31:04.383950 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:04.383929    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"
Jul 20 10:31:04.384134 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:04.383953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z" event={"ID":"597e2c52-5f4f-43de-81c7-f530332222cf","Type":"ContainerStarted","Data":"d5ade355ae52513a9017fc98822116ad661967e3a41687dec26c902ff2bddf52"}
Jul 20 10:31:10.825388 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:10.825352    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:31:35.390901 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:35.390869    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"
Jul 20 10:31:35.408269 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:35.408214    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z" podStartSLOduration=32.408195579 podStartE2EDuration="32.408195579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 10:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:31:04.404251357 +0000 UTC m=+2076.453115007" watchObservedRunningTime="2026-07-20 10:31:35.408195579 +0000 UTC m=+2107.457059219"
Jul 20 10:31:44.998473 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:44.998442    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:31:53.950457 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:31:53.950425    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:32:24.887437 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:32:24.887403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:33:59.629217 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:33:59.629111    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"]
Jul 20 10:33:59.660049 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:33:59.659996    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:01.037611 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:01.037570    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:01.404841 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:01.404742    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:01.473837 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:01.473807    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:05.391282 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:05.391243    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z" podUID="597e2c52-5f4f-43de-81c7-f530332222cf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:34:14.650622 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:14.650564    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z" podUID="597e2c52-5f4f-43de-81c7-f530332222cf" containerName="main" containerID="cri-o://b833abb2de2b3526a1164f5d4995444c2fc4514d34af23d95558fb9ddcee0c53" gracePeriod=45
Jul 20 10:34:14.894547 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:14.894520    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"
Jul 20 10:34:15.064123 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:15.064082    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/597e2c52-5f4f-43de-81c7-f530332222cf-tls-certs\") pod \"597e2c52-5f4f-43de-81c7-f530332222cf\" (UID: \"597e2c52-5f4f-43de-81c7-f530332222cf\") "
Jul 20 10:34:15.064366 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:15.064137    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/597e2c52-5f4f-43de-81c7-f530332222cf-tokenizer-uds\") pod \"597e2c52-5f4f-43de-81c7-f530332222cf\" (UID: \"597e2c52-5f4f-43de-81c7-f530332222cf\") "
Jul 20 10:34:15.064366 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:15.064224    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rszzn\" (UniqueName: \"kubernetes.io/projected/597e2c52-5f4f-43de-81c7-f530332222cf-kube-api-access-rszzn\") pod \"597e2c52-5f4f-43de-81c7-f530332222cf\" (UID: \"597e2c52-5f4f-43de-81c7-f530332222cf\") "
Jul 20 10:34:15.064507 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:15.064486    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/597e2c52-5f4f-43de-81c7-f530332222cf-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "597e2c52-5f4f-43de-81c7-f530332222cf" (UID: "597e2c52-5f4f-43de-81c7-f530332222cf"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:34:15.066654 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:15.066628    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/597e2c52-5f4f-43de-81c7-f530332222cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "597e2c52-5f4f-43de-81c7-f530332222cf" (UID: "597e2c52-5f4f-43de-81c7-f530332222cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:34:15.066775 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:15.066658    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/597e2c52-5f4f-43de-81c7-f530332222cf-kube-api-access-rszzn" (OuterVolumeSpecName: "kube-api-access-rszzn") pod "597e2c52-5f4f-43de-81c7-f530332222cf" (UID: "597e2c52-5f4f-43de-81c7-f530332222cf"). InnerVolumeSpecName "kube-api-access-rszzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:34:15.164936 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:15.164896    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rszzn\" (UniqueName: \"kubernetes.io/projected/597e2c52-5f4f-43de-81c7-f530332222cf-kube-api-access-rszzn\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:34:15.164936 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:15.164931    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/597e2c52-5f4f-43de-81c7-f530332222cf-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:34:15.164936 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:15.164942    2580 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/597e2c52-5f4f-43de-81c7-f530332222cf-tokenizer-uds\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:34:15.378315 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:15.378221    2580 generic.go:356] "Generic (PLEG): container finished" podID="597e2c52-5f4f-43de-81c7-f530332222cf" containerID="b833abb2de2b3526a1164f5d4995444c2fc4514d34af23d95558fb9ddcee0c53" exitCode=0
Jul 20 10:34:15.378315 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:15.378285    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"
Jul 20 10:34:15.378545 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:15.378307    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z" event={"ID":"597e2c52-5f4f-43de-81c7-f530332222cf","Type":"ContainerDied","Data":"b833abb2de2b3526a1164f5d4995444c2fc4514d34af23d95558fb9ddcee0c53"}
Jul 20 10:34:15.378545 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:15.378362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z" event={"ID":"597e2c52-5f4f-43de-81c7-f530332222cf","Type":"ContainerDied","Data":"d5ade355ae52513a9017fc98822116ad661967e3a41687dec26c902ff2bddf52"}
Jul 20 10:34:15.378545 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:15.378386    2580 scope.go:117] "RemoveContainer" containerID="b833abb2de2b3526a1164f5d4995444c2fc4514d34af23d95558fb9ddcee0c53"
Jul 20 10:34:15.387712 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:15.387694    2580 scope.go:117] "RemoveContainer" containerID="b833abb2de2b3526a1164f5d4995444c2fc4514d34af23d95558fb9ddcee0c53"
Jul 20 10:34:15.387998 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:34:15.387979    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b833abb2de2b3526a1164f5d4995444c2fc4514d34af23d95558fb9ddcee0c53\": container with ID starting with b833abb2de2b3526a1164f5d4995444c2fc4514d34af23d95558fb9ddcee0c53 not found: ID does not exist" containerID="b833abb2de2b3526a1164f5d4995444c2fc4514d34af23d95558fb9ddcee0c53"
Jul 20 10:34:15.388051 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:15.388008    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b833abb2de2b3526a1164f5d4995444c2fc4514d34af23d95558fb9ddcee0c53"} err="failed to get container status \"b833abb2de2b3526a1164f5d4995444c2fc4514d34af23d95558fb9ddcee0c53\": rpc error: code = NotFound desc = could not find container \"b833abb2de2b3526a1164f5d4995444c2fc4514d34af23d95558fb9ddcee0c53\": container with ID starting with b833abb2de2b3526a1164f5d4995444c2fc4514d34af23d95558fb9ddcee0c53 not found: ID does not exist"
Jul 20 10:34:15.399287 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:15.399261    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"]
Jul 20 10:34:15.402208 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:15.402188    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqc56z"]
Jul 20 10:34:16.410609 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:16.410574    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="597e2c52-5f4f-43de-81c7-f530332222cf" path="/var/lib/kubelet/pods/597e2c52-5f4f-43de-81c7-f530332222cf/volumes"
Jul 20 10:34:19.737058 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:19.737027    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:19.834247 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:19.834212    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:19.935432 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:19.935395    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:20.033793 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:20.033696    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:20.133635 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:20.133603    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:22.288494 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:22.288460    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:22.413402 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:22.413370    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:23.119368 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:23.119316    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:26.396144 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:26.396105    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:26.494303 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:26.494276    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:26.706499 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:26.706465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:32.497568 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:32.497512    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:32.599292 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:32.599257    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:33.644533 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:33.644497    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:33.751253 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:33.751212    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:42.274063 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:42.274028    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:42.399489 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:42.399446    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:42.577756 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:42.577665    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:42.850576 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:42.850495    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"]
Jul 20 10:34:42.850838 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:42.850826    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="597e2c52-5f4f-43de-81c7-f530332222cf" containerName="main"
Jul 20 10:34:42.850880 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:42.850840    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="597e2c52-5f4f-43de-81c7-f530332222cf" containerName="main"
Jul 20 10:34:42.851053 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:42.850896    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="597e2c52-5f4f-43de-81c7-f530332222cf" containerName="main"
Jul 20 10:34:42.853895 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:42.853873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"
Jul 20 10:34:42.857881 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:42.857858    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 20 10:34:42.858001 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:42.857869    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 20 10:34:42.858001 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:42.857857    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-s5hvh\""
Jul 20 10:34:42.858332 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:42.858315    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 20 10:34:42.858431 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:42.858317    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-wjp7b\""
Jul 20 10:34:42.870680 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:42.870651    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"]
Jul 20 10:34:42.901655 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:42.901612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whzl7\" (UniqueName: \"kubernetes.io/projected/428668e7-b6c3-4e6b-ad4c-2f6727628a21-kube-api-access-whzl7\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx\" (UID: \"428668e7-b6c3-4e6b-ad4c-2f6727628a21\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"
Jul 20 10:34:42.901824 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:42.901685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/428668e7-b6c3-4e6b-ad4c-2f6727628a21-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx\" (UID: \"428668e7-b6c3-4e6b-ad4c-2f6727628a21\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"
Jul 20 10:34:42.901824 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:42.901742    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/428668e7-b6c3-4e6b-ad4c-2f6727628a21-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx\" (UID: \"428668e7-b6c3-4e6b-ad4c-2f6727628a21\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"
Jul 20 10:34:42.905308 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:42.905274    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:34:43.002554 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:43.002519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/428668e7-b6c3-4e6b-ad4c-2f6727628a21-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx\" (UID: \"428668e7-b6c3-4e6b-ad4c-2f6727628a21\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"
Jul 20 10:34:43.002760 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:43.002597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whzl7\" (UniqueName: \"kubernetes.io/projected/428668e7-b6c3-4e6b-ad4c-2f6727628a21-kube-api-access-whzl7\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx\" (UID: \"428668e7-b6c3-4e6b-ad4c-2f6727628a21\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"
Jul 20 10:34:43.002760 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:43.002643    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/428668e7-b6c3-4e6b-ad4c-2f6727628a21-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx\" (UID: \"428668e7-b6c3-4e6b-ad4c-2f6727628a21\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"
Jul 20 10:34:43.003013 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:43.002997    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/428668e7-b6c3-4e6b-ad4c-2f6727628a21-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx\" (UID: \"428668e7-b6c3-4e6b-ad4c-2f6727628a21\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"
Jul 20 10:34:43.005211 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:43.005188    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/428668e7-b6c3-4e6b-ad4c-2f6727628a21-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx\" (UID: \"428668e7-b6c3-4e6b-ad4c-2f6727628a21\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"
Jul 20 10:34:43.012634 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:43.012612    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whzl7\" (UniqueName: \"kubernetes.io/projected/428668e7-b6c3-4e6b-ad4c-2f6727628a21-kube-api-access-whzl7\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx\" (UID: \"428668e7-b6c3-4e6b-ad4c-2f6727628a21\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"
Jul 20 10:34:43.164238 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:43.164127    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"
Jul 20 10:34:43.315668 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:43.315639    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"]
Jul 20 10:34:43.321479 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:34:43.321451    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428668e7_b6c3_4e6b_ad4c_2f6727628a21.slice/crio-3ed37123d3c91a51037ae82077b8e4adabed364220ec1ca7a7e1cfcda22aaf69 WatchSource:0}: Error finding container 3ed37123d3c91a51037ae82077b8e4adabed364220ec1ca7a7e1cfcda22aaf69: Status 404 returned error can't find the container with id 3ed37123d3c91a51037ae82077b8e4adabed364220ec1ca7a7e1cfcda22aaf69
Jul 20 10:34:43.520804 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:43.520772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx" event={"ID":"428668e7-b6c3-4e6b-ad4c-2f6727628a21","Type":"ContainerStarted","Data":"f59784310f183ec07b8c1cdf33c4b590b953964dd5d17c56a7f2f12da0c72507"}
Jul 20 10:34:43.520998 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:43.520834    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx" event={"ID":"428668e7-b6c3-4e6b-ad4c-2f6727628a21","Type":"ContainerStarted","Data":"3ed37123d3c91a51037ae82077b8e4adabed364220ec1ca7a7e1cfcda22aaf69"}
Jul 20 10:34:43.520998 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:43.520860    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"
Jul 20 10:34:43.563534 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:43.563479    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx" podStartSLOduration=1.5634630120000002 podStartE2EDuration="1.563463012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 10:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:34:43.561900464 +0000 UTC m=+2295.610764113" watchObservedRunningTime="2026-07-20 10:34:43.563463012 +0000 UTC m=+2295.612326658"
Jul 20 10:34:57.524456 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:34:57.524422    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:35:14.528790 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:35:14.528760    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"
Jul 20 10:37:02.332831 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:37:02.332750    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:39:16.912743 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:16.912711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:39:21.871298 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:21.871260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:39:21.971983 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:21.971951    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:39:22.084186 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:22.084117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:39:22.727221 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:22.727185    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:39:22.764643 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:22.764607    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"]
Jul 20 10:39:22.824062 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:22.824027    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:39:24.527901 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:24.527854    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx" podUID="428668e7-b6c3-4e6b-ad4c-2f6727628a21" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:39:28.043513 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:28.043474    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:39:29.442013 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:29.441976    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:39:33.709699 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:33.709662    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:39:33.813025 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:33.812988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:39:34.528464 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:34.528424    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx" podUID="428668e7-b6c3-4e6b-ad4c-2f6727628a21" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:39:37.785987 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:37.785941    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx" podUID="428668e7-b6c3-4e6b-ad4c-2f6727628a21" containerName="main" containerID="cri-o://f59784310f183ec07b8c1cdf33c4b590b953964dd5d17c56a7f2f12da0c72507" gracePeriod=45
Jul 20 10:39:38.024966 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:38.024939    2580 generic.go:356] "Generic (PLEG): container finished" podID="428668e7-b6c3-4e6b-ad4c-2f6727628a21" containerID="f59784310f183ec07b8c1cdf33c4b590b953964dd5d17c56a7f2f12da0c72507" exitCode=0
Jul 20 10:39:38.025108 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:38.025017    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx" event={"ID":"428668e7-b6c3-4e6b-ad4c-2f6727628a21","Type":"ContainerDied","Data":"f59784310f183ec07b8c1cdf33c4b590b953964dd5d17c56a7f2f12da0c72507"}
Jul 20 10:39:38.025108 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:38.025096    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx" event={"ID":"428668e7-b6c3-4e6b-ad4c-2f6727628a21","Type":"ContainerDied","Data":"3ed37123d3c91a51037ae82077b8e4adabed364220ec1ca7a7e1cfcda22aaf69"}
Jul 20 10:39:38.025223 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:38.025113    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ed37123d3c91a51037ae82077b8e4adabed364220ec1ca7a7e1cfcda22aaf69"
Jul 20 10:39:38.037875 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:38.037803    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"
Jul 20 10:39:38.084753 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:38.084722    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/428668e7-b6c3-4e6b-ad4c-2f6727628a21-tokenizer-uds\") pod \"428668e7-b6c3-4e6b-ad4c-2f6727628a21\" (UID: \"428668e7-b6c3-4e6b-ad4c-2f6727628a21\") "
Jul 20 10:39:38.084934 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:38.084760    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whzl7\" (UniqueName: \"kubernetes.io/projected/428668e7-b6c3-4e6b-ad4c-2f6727628a21-kube-api-access-whzl7\") pod \"428668e7-b6c3-4e6b-ad4c-2f6727628a21\" (UID: \"428668e7-b6c3-4e6b-ad4c-2f6727628a21\") "
Jul 20 10:39:38.084934 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:38.084796    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/428668e7-b6c3-4e6b-ad4c-2f6727628a21-tls-certs\") pod \"428668e7-b6c3-4e6b-ad4c-2f6727628a21\" (UID: \"428668e7-b6c3-4e6b-ad4c-2f6727628a21\") "
Jul 20 10:39:38.085028 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:38.085007    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428668e7-b6c3-4e6b-ad4c-2f6727628a21-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "428668e7-b6c3-4e6b-ad4c-2f6727628a21" (UID: "428668e7-b6c3-4e6b-ad4c-2f6727628a21"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:39:38.087125 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:38.087099    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428668e7-b6c3-4e6b-ad4c-2f6727628a21-kube-api-access-whzl7" (OuterVolumeSpecName: "kube-api-access-whzl7") pod "428668e7-b6c3-4e6b-ad4c-2f6727628a21" (UID: "428668e7-b6c3-4e6b-ad4c-2f6727628a21"). InnerVolumeSpecName "kube-api-access-whzl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:39:38.087125 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:38.087098    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/428668e7-b6c3-4e6b-ad4c-2f6727628a21-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "428668e7-b6c3-4e6b-ad4c-2f6727628a21" (UID: "428668e7-b6c3-4e6b-ad4c-2f6727628a21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:39:38.185616 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:38.185570    2580 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/428668e7-b6c3-4e6b-ad4c-2f6727628a21-tokenizer-uds\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:39:38.185616 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:38.185610    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whzl7\" (UniqueName: \"kubernetes.io/projected/428668e7-b6c3-4e6b-ad4c-2f6727628a21-kube-api-access-whzl7\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:39:38.185616 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:38.185624    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/428668e7-b6c3-4e6b-ad4c-2f6727628a21-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:39:39.030259 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:39.030224    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"
Jul 20 10:39:39.049864 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:39.049833    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"]
Jul 20 10:39:39.051782 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:39.051759    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheplrkx"]
Jul 20 10:39:40.411107 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:40.411070    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="428668e7-b6c3-4e6b-ad4c-2f6727628a21" path="/var/lib/kubelet/pods/428668e7-b6c3-4e6b-ad4c-2f6727628a21/volumes"
Jul 20 10:39:53.578922 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:53.578834    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:39:57.613542 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:57.613505    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:39:57.714296 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:57.714253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:39:57.813851 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:57.813818    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:39:59.874849 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:59.874814    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:39:59.978449 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:39:59.978422    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:00.078455 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:00.078421    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:03.515123 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:03.515085    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:03.615600 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:03.615562    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:03.994888 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:03.994855    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:04.855477 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:04.855441    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:04.915517 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:04.915477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:05.615410 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:05.615379    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:05.676024 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:05.675995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:07.957748 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:07.957720    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:08.359037 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:08.358947    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:10.119278 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:10.119222    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:10.222377 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:10.222326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:11.141304 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:11.141239    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:11.220195 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:11.219212    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:12.440148 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:12.440106    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:12.579795 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:12.579753    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:12.709723 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:12.709665    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:16.844195 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:16.844105    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:16.948131 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:16.948087    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:17.052071 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:17.051982    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:17.512225 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:17.512151    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:19.325361 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:19.325324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:19.513071 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:19.513040    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:19.621145 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:19.621057    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:22.561223 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:22.561152    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:22.665288 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:22.665253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:25.093336 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:25.093298    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:25.194098 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:25.194062    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:28.213528 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.213495    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:28.425336 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.425296    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"]
Jul 20 10:40:28.425801 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.425782    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="428668e7-b6c3-4e6b-ad4c-2f6727628a21" containerName="main"
Jul 20 10:40:28.425873 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.425806    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="428668e7-b6c3-4e6b-ad4c-2f6727628a21" containerName="main"
Jul 20 10:40:28.425936 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.425906    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="428668e7-b6c3-4e6b-ad4c-2f6727628a21" containerName="main"
Jul 20 10:40:28.429528 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.429496    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"
Jul 20 10:40:28.435292 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.435256    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-qs5vk\""
Jul 20 10:40:28.435427 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.435293    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Jul 20 10:40:28.435427 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.435324    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Jul 20 10:40:28.435427 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.435293    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-98cfd\""
Jul 20 10:40:28.435427 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.435256    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-kserve-self-signed-certs\""
Jul 20 10:40:28.435740 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.435724    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"seaweedfs-s3-creds\""
Jul 20 10:40:28.443612 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.443589    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"]
Jul 20 10:40:28.462848 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.462816    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:28.525466 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.525378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/86c28ace-0727-4fbc-8890-0693d5424699-tokenizer-uds\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b\" (UID: \"86c28ace-0727-4fbc-8890-0693d5424699\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"
Jul 20 10:40:28.525466 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.525423    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86c28ace-0727-4fbc-8890-0693d5424699-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b\" (UID: \"86c28ace-0727-4fbc-8890-0693d5424699\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"
Jul 20 10:40:28.525466 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.525457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c28ace-0727-4fbc-8890-0693d5424699-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b\" (UID: \"86c28ace-0727-4fbc-8890-0693d5424699\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"
Jul 20 10:40:28.525703 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.525499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj7lb\" (UniqueName: \"kubernetes.io/projected/86c28ace-0727-4fbc-8890-0693d5424699-kube-api-access-rj7lb\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b\" (UID: \"86c28ace-0727-4fbc-8890-0693d5424699\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"
Jul 20 10:40:28.626724 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.626682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj7lb\" (UniqueName: \"kubernetes.io/projected/86c28ace-0727-4fbc-8890-0693d5424699-kube-api-access-rj7lb\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b\" (UID: \"86c28ace-0727-4fbc-8890-0693d5424699\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"
Jul 20 10:40:28.626937 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.626773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/86c28ace-0727-4fbc-8890-0693d5424699-tokenizer-uds\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b\" (UID: \"86c28ace-0727-4fbc-8890-0693d5424699\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"
Jul 20 10:40:28.626937 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.626810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86c28ace-0727-4fbc-8890-0693d5424699-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b\" (UID: \"86c28ace-0727-4fbc-8890-0693d5424699\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"
Jul 20 10:40:28.626937 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.626851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c28ace-0727-4fbc-8890-0693d5424699-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b\" (UID: \"86c28ace-0727-4fbc-8890-0693d5424699\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"
Jul 20 10:40:28.627280 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.627257    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/86c28ace-0727-4fbc-8890-0693d5424699-tokenizer-uds\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b\" (UID: \"86c28ace-0727-4fbc-8890-0693d5424699\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"
Jul 20 10:40:28.627280 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.627271    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c28ace-0727-4fbc-8890-0693d5424699-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b\" (UID: \"86c28ace-0727-4fbc-8890-0693d5424699\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"
Jul 20 10:40:28.629454 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.629437    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86c28ace-0727-4fbc-8890-0693d5424699-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b\" (UID: \"86c28ace-0727-4fbc-8890-0693d5424699\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"
Jul 20 10:40:28.634411 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.634387    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj7lb\" (UniqueName: \"kubernetes.io/projected/86c28ace-0727-4fbc-8890-0693d5424699-kube-api-access-rj7lb\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b\" (UID: \"86c28ace-0727-4fbc-8890-0693d5424699\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"
Jul 20 10:40:28.742323 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.742288    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"
Jul 20 10:40:28.883602 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:28.883567    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"]
Jul 20 10:40:28.885898 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:40:28.885866    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c28ace_0727_4fbc_8890_0693d5424699.slice/crio-306436145aab399c14f4627dacf8068b41eeb2540a25142f11d0e7bae0901777 WatchSource:0}: Error finding container 306436145aab399c14f4627dacf8068b41eeb2540a25142f11d0e7bae0901777: Status 404 returned error can't find the container with id 306436145aab399c14f4627dacf8068b41eeb2540a25142f11d0e7bae0901777
Jul 20 10:40:29.295610 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:29.295566    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b" event={"ID":"86c28ace-0727-4fbc-8890-0693d5424699","Type":"ContainerStarted","Data":"4b42d0d9d826c619fa4d555e0e60bcff94558df5193bb6ba8b659109d202656b"}
Jul 20 10:40:29.295610 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:29.295620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b" event={"ID":"86c28ace-0727-4fbc-8890-0693d5424699","Type":"ContainerStarted","Data":"306436145aab399c14f4627dacf8068b41eeb2540a25142f11d0e7bae0901777"}
Jul 20 10:40:30.301860 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:30.301825    2580 generic.go:356] "Generic (PLEG): container finished" podID="86c28ace-0727-4fbc-8890-0693d5424699" containerID="4b42d0d9d826c619fa4d555e0e60bcff94558df5193bb6ba8b659109d202656b" exitCode=0
Jul 20 10:40:30.302276 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:30.301914    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b" event={"ID":"86c28ace-0727-4fbc-8890-0693d5424699","Type":"ContainerDied","Data":"4b42d0d9d826c619fa4d555e0e60bcff94558df5193bb6ba8b659109d202656b"}
Jul 20 10:40:31.290352 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:31.290315    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"]
Jul 20 10:40:31.291837 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:31.291804    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:31.309471 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:31.309440    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b" event={"ID":"86c28ace-0727-4fbc-8890-0693d5424699","Type":"ContainerStarted","Data":"c327dc6ba9fbb72cf14df6894cfb42cc0b64bc9153d636c2ae765961ebcf8f38"}
Jul 20 10:40:31.309976 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:31.309518    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"
Jul 20 10:40:31.309976 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:31.309571    2580 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b" secret="" err="secret \"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-98cfd\" not found"
Jul 20 10:40:31.330144 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:31.330080    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b" podStartSLOduration=2.622358325 podStartE2EDuration="3.330059959s" totalImagesPullingTime="0s" totalInitContainerRuntime="707.701634ms" isStatefulPod=false podCreationTimestamp="2026-07-20 10:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:40:31.327400789 +0000 UTC m=+2643.376264437" watchObservedRunningTime="2026-07-20 10:40:31.330059959 +0000 UTC m=+2643.378923608"
Jul 20 10:40:31.352515 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:31.352487    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:31.451367 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:40:31.451332    2580 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Jul 20 10:40:31.451549 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:40:31.451426    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86c28ace-0727-4fbc-8890-0693d5424699-tls-certs podName:86c28ace-0727-4fbc-8890-0693d5424699 nodeName:}" failed. No retries permitted until 2026-07-20 10:40:31.951409723 +0000 UTC m=+2644.000273348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/86c28ace-0727-4fbc-8890-0693d5424699-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b" (UID: "86c28ace-0727-4fbc-8890-0693d5424699") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Jul 20 10:40:31.955569 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:40:31.955537    2580 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Jul 20 10:40:31.955732 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:40:31.955604    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86c28ace-0727-4fbc-8890-0693d5424699-tls-certs podName:86c28ace-0727-4fbc-8890-0693d5424699 nodeName:}" failed. No retries permitted until 2026-07-20 10:40:32.95558921 +0000 UTC m=+2645.004452835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/86c28ace-0727-4fbc-8890-0693d5424699-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b" (UID: "86c28ace-0727-4fbc-8890-0693d5424699") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Jul 20 10:40:32.964776 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:40:32.964737    2580 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Jul 20 10:40:32.965231 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:40:32.964819    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86c28ace-0727-4fbc-8890-0693d5424699-tls-certs podName:86c28ace-0727-4fbc-8890-0693d5424699 nodeName:}" failed. No retries permitted until 2026-07-20 10:40:34.964800208 +0000 UTC m=+2647.013663838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/86c28ace-0727-4fbc-8890-0693d5424699-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b" (UID: "86c28ace-0727-4fbc-8890-0693d5424699") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Jul 20 10:40:33.944652 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:33.944613    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:34.048892 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:34.048860    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:34.146632 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:34.146597    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:34.632682 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:34.632641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:34.731557 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:34.731522    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:34.832927 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:34.832869    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:34.979399 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:40:34.979368    2580 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Jul 20 10:40:34.979585 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:40:34.979436    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86c28ace-0727-4fbc-8890-0693d5424699-tls-certs podName:86c28ace-0727-4fbc-8890-0693d5424699 nodeName:}" failed. No retries permitted until 2026-07-20 10:40:38.97942157 +0000 UTC m=+2651.028285195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/86c28ace-0727-4fbc-8890-0693d5424699-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b" (UID: "86c28ace-0727-4fbc-8890-0693d5424699") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Jul 20 10:40:35.324627 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:35.324529    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:35.425607 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:35.425580    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:35.526028 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:35.525994    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:36.723548 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:36.723473    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:36.851929 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:36.851895    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:36.952969 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:36.952934    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:37.052661 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:37.052539    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:37.922431 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:37.922385    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:38.055704 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:38.055666    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:38.148132 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:38.148099    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:39.020901 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:40:39.020866    2580 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Jul 20 10:40:39.021374 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:40:39.020953    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86c28ace-0727-4fbc-8890-0693d5424699-tls-certs podName:86c28ace-0727-4fbc-8890-0693d5424699 nodeName:}" failed. No retries permitted until 2026-07-20 10:40:47.020933908 +0000 UTC m=+2659.069797536 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/86c28ace-0727-4fbc-8890-0693d5424699-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b" (UID: "86c28ace-0727-4fbc-8890-0693d5424699") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Jul 20 10:40:39.129159 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:39.129115    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:39.502324 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:39.502289    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:39.603472 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:39.603437    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:41.533322 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:41.533289    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:42.187301 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:42.187267    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:42.257059 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:42.257019    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:42.864822 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:42.864784    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:42.947324 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:42.947286    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:44.060883 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:44.060854    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:44.135053 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:44.135025    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:45.816353 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:45.816320    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:45.882474 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:45.882438    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:47.098611 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:40:47.098561    2580 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Jul 20 10:40:47.099026 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:40:47.098647    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86c28ace-0727-4fbc-8890-0693d5424699-tls-certs podName:86c28ace-0727-4fbc-8890-0693d5424699 nodeName:}" failed. No retries permitted until 2026-07-20 10:41:03.098632462 +0000 UTC m=+2675.147496087 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/86c28ace-0727-4fbc-8890-0693d5424699-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b" (UID: "86c28ace-0727-4fbc-8890-0693d5424699") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Jul 20 10:40:47.337989 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:47.337952    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b" podUID="86c28ace-0727-4fbc-8890-0693d5424699" containerName="main" containerID="cri-o://c327dc6ba9fbb72cf14df6894cfb42cc0b64bc9153d636c2ae765961ebcf8f38" gracePeriod=45
Jul 20 10:40:47.593729 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:47.593705    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"
Jul 20 10:40:47.703695 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:47.703662    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/86c28ace-0727-4fbc-8890-0693d5424699-tokenizer-uds\") pod \"86c28ace-0727-4fbc-8890-0693d5424699\" (UID: \"86c28ace-0727-4fbc-8890-0693d5424699\") "
Jul 20 10:40:47.703885 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:47.703813    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c28ace-0727-4fbc-8890-0693d5424699-kserve-provision-location\") pod \"86c28ace-0727-4fbc-8890-0693d5424699\" (UID: \"86c28ace-0727-4fbc-8890-0693d5424699\") "
Jul 20 10:40:47.703885 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:47.703846    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rj7lb\" (UniqueName: \"kubernetes.io/projected/86c28ace-0727-4fbc-8890-0693d5424699-kube-api-access-rj7lb\") pod \"86c28ace-0727-4fbc-8890-0693d5424699\" (UID: \"86c28ace-0727-4fbc-8890-0693d5424699\") "
Jul 20 10:40:47.704006 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:47.703889    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86c28ace-0727-4fbc-8890-0693d5424699-tls-certs\") pod \"86c28ace-0727-4fbc-8890-0693d5424699\" (UID: \"86c28ace-0727-4fbc-8890-0693d5424699\") "
Jul 20 10:40:47.704006 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:47.703969    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c28ace-0727-4fbc-8890-0693d5424699-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "86c28ace-0727-4fbc-8890-0693d5424699" (UID: "86c28ace-0727-4fbc-8890-0693d5424699"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:40:47.704187 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:47.704146    2580 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/86c28ace-0727-4fbc-8890-0693d5424699-tokenizer-uds\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:40:47.704686 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:47.704652    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c28ace-0727-4fbc-8890-0693d5424699-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86c28ace-0727-4fbc-8890-0693d5424699" (UID: "86c28ace-0727-4fbc-8890-0693d5424699"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:40:47.706825 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:47.706762    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c28ace-0727-4fbc-8890-0693d5424699-kube-api-access-rj7lb" (OuterVolumeSpecName: "kube-api-access-rj7lb") pod "86c28ace-0727-4fbc-8890-0693d5424699" (UID: "86c28ace-0727-4fbc-8890-0693d5424699"). InnerVolumeSpecName "kube-api-access-rj7lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:40:47.706825 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:47.706782    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86c28ace-0727-4fbc-8890-0693d5424699-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86c28ace-0727-4fbc-8890-0693d5424699" (UID: "86c28ace-0727-4fbc-8890-0693d5424699"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:40:47.750877 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:47.750841    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:47.805230 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:47.805196    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c28ace-0727-4fbc-8890-0693d5424699-kserve-provision-location\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:40:47.805230 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:47.805226    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj7lb\" (UniqueName: \"kubernetes.io/projected/86c28ace-0727-4fbc-8890-0693d5424699-kube-api-access-rj7lb\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:40:47.805230 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:47.805237    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86c28ace-0727-4fbc-8890-0693d5424699-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:40:47.966402 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:47.966313    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:48.395880 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:48.395795    2580 generic.go:356] "Generic (PLEG): container finished" podID="86c28ace-0727-4fbc-8890-0693d5424699" containerID="c327dc6ba9fbb72cf14df6894cfb42cc0b64bc9153d636c2ae765961ebcf8f38" exitCode=0
Jul 20 10:40:48.395880 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:48.395845    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b" event={"ID":"86c28ace-0727-4fbc-8890-0693d5424699","Type":"ContainerDied","Data":"c327dc6ba9fbb72cf14df6894cfb42cc0b64bc9153d636c2ae765961ebcf8f38"}
Jul 20 10:40:48.395880 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:48.395854    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"
Jul 20 10:40:48.396451 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:48.395888    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b" event={"ID":"86c28ace-0727-4fbc-8890-0693d5424699","Type":"ContainerDied","Data":"306436145aab399c14f4627dacf8068b41eeb2540a25142f11d0e7bae0901777"}
Jul 20 10:40:48.396451 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:48.395909    2580 scope.go:117] "RemoveContainer" containerID="c327dc6ba9fbb72cf14df6894cfb42cc0b64bc9153d636c2ae765961ebcf8f38"
Jul 20 10:40:48.405767 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:48.405749    2580 scope.go:117] "RemoveContainer" containerID="4b42d0d9d826c619fa4d555e0e60bcff94558df5193bb6ba8b659109d202656b"
Jul 20 10:40:48.414376 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:48.414356    2580 scope.go:117] "RemoveContainer" containerID="c327dc6ba9fbb72cf14df6894cfb42cc0b64bc9153d636c2ae765961ebcf8f38"
Jul 20 10:40:48.414611 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:40:48.414591    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c327dc6ba9fbb72cf14df6894cfb42cc0b64bc9153d636c2ae765961ebcf8f38\": container with ID starting with c327dc6ba9fbb72cf14df6894cfb42cc0b64bc9153d636c2ae765961ebcf8f38 not found: ID does not exist" containerID="c327dc6ba9fbb72cf14df6894cfb42cc0b64bc9153d636c2ae765961ebcf8f38"
Jul 20 10:40:48.414678 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:48.414621    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c327dc6ba9fbb72cf14df6894cfb42cc0b64bc9153d636c2ae765961ebcf8f38"} err="failed to get container status \"c327dc6ba9fbb72cf14df6894cfb42cc0b64bc9153d636c2ae765961ebcf8f38\": rpc error: code = NotFound desc = could not find container \"c327dc6ba9fbb72cf14df6894cfb42cc0b64bc9153d636c2ae765961ebcf8f38\": container with ID starting with c327dc6ba9fbb72cf14df6894cfb42cc0b64bc9153d636c2ae765961ebcf8f38 not found: ID does not exist"
Jul 20 10:40:48.414678 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:48.414638    2580 scope.go:117] "RemoveContainer" containerID="4b42d0d9d826c619fa4d555e0e60bcff94558df5193bb6ba8b659109d202656b"
Jul 20 10:40:48.414869 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:40:48.414852    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b42d0d9d826c619fa4d555e0e60bcff94558df5193bb6ba8b659109d202656b\": container with ID starting with 4b42d0d9d826c619fa4d555e0e60bcff94558df5193bb6ba8b659109d202656b not found: ID does not exist" containerID="4b42d0d9d826c619fa4d555e0e60bcff94558df5193bb6ba8b659109d202656b"
Jul 20 10:40:48.414910 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:48.414878    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b42d0d9d826c619fa4d555e0e60bcff94558df5193bb6ba8b659109d202656b"} err="failed to get container status \"4b42d0d9d826c619fa4d555e0e60bcff94558df5193bb6ba8b659109d202656b\": rpc error: code = NotFound desc = could not find container \"4b42d0d9d826c619fa4d555e0e60bcff94558df5193bb6ba8b659109d202656b\": container with ID starting with 4b42d0d9d826c619fa4d555e0e60bcff94558df5193bb6ba8b659109d202656b not found: ID does not exist"
Jul 20 10:40:48.419048 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:48.419028    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"]
Jul 20 10:40:48.431334 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:48.431309    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-5cb48c5b4dm9j7b"]
Jul 20 10:40:50.410697 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:50.410667    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86c28ace-0727-4fbc-8890-0693d5424699" path="/var/lib/kubelet/pods/86c28ace-0727-4fbc-8890-0693d5424699/volumes"
Jul 20 10:40:54.317458 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:54.317424    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:56.014277 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:56.014236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:56.117065 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:40:56.117026    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:41:00.904933 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:41:00.904893    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:41:16.014467 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:41:16.014412    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:41:25.951011 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:41:25.950919    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:41:28.671184 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:41:28.671147    2580 scope.go:117] "RemoveContainer" containerID="f59784310f183ec07b8c1cdf33c4b590b953964dd5d17c56a7f2f12da0c72507"
Jul 20 10:42:49.417782 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:42:49.417700    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:42:49.720756 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:42:49.720722    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:42:49.823951 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:42:49.823917    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:43:04.916243 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:43:04.916210    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:43:21.934428 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:43:21.934385    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:43:22.229774 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:43:22.229739    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:43:31.249853 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:43:31.249783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:45:23.896853 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:45:23.896804    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:45:25.024833 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:45:25.024797    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:45:43.208086 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:45:43.208052    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:45:43.934102 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:45:43.934068    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:45:44.036216 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:45:44.036156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:45:49.377639 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:45:49.377557    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:45:53.939124 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:45:53.939098    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:45:55.361778 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:45:55.361744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:45:55.469744 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:45:55.469711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:46:44.099544 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:46:44.099506    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:47:50.752257 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:47:50.752221    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:48:10.897915 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:48:10.897883    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:48:11.001507 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:48:11.001474    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:48:11.101329 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:48:11.101290    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:48:12.181646 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:48:12.181608    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:48:16.716151 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:48:16.716117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:48:16.816416 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:48:16.816382    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:48:28.438507 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:48:28.438471    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:49:19.383699 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:49:19.383613    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:49:19.440693 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:49:19.440660    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:50:36.282777 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:50:36.282740    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:50:56.547263 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:50:56.547233    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:50:56.649862 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:50:56.649828    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:51:02.462305 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:02.462271    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:51:02.529238 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:02.529209    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:51:08.739159 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:08.739119    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:51:08.941355 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:08.941323    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"]
Jul 20 10:51:08.941652 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:08.941639    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c28ace-0727-4fbc-8890-0693d5424699" containerName="storage-initializer"
Jul 20 10:51:08.941705 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:08.941653    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c28ace-0727-4fbc-8890-0693d5424699" containerName="storage-initializer"
Jul 20 10:51:08.941705 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:08.941675    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c28ace-0727-4fbc-8890-0693d5424699" containerName="main"
Jul 20 10:51:08.941705 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:08.941681    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c28ace-0727-4fbc-8890-0693d5424699" containerName="main"
Jul 20 10:51:08.941830 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:08.941752    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="86c28ace-0727-4fbc-8890-0693d5424699" containerName="main"
Jul 20 10:51:08.944758 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:08.944735    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"
Jul 20 10:51:08.946929 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:08.946906    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Jul 20 10:51:08.947073 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:08.946911    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Jul 20 10:51:08.947073 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:08.946989    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-c6fxh\""
Jul 20 10:51:08.947073 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:08.947009    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Jul 20 10:51:08.947755 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:08.947739    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-9jpwk\""
Jul 20 10:51:08.947810 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:08.947799    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 20 10:51:08.954716 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:08.954696    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"]
Jul 20 10:51:08.965910 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:08.965886    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:51:09.030779 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:09.030685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42wm9\" (UniqueName: \"kubernetes.io/projected/8f4a9891-73f7-4412-97a6-d1cae0a5f902-kube-api-access-42wm9\") pod \"tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw\" (UID: \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"
Jul 20 10:51:09.030779 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:09.030737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8f4a9891-73f7-4412-97a6-d1cae0a5f902-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw\" (UID: \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"
Jul 20 10:51:09.030977 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:09.030784    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f4a9891-73f7-4412-97a6-d1cae0a5f902-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw\" (UID: \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"
Jul 20 10:51:09.030977 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:09.030811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f4a9891-73f7-4412-97a6-d1cae0a5f902-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw\" (UID: \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"
Jul 20 10:51:09.132069 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:09.132035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42wm9\" (UniqueName: \"kubernetes.io/projected/8f4a9891-73f7-4412-97a6-d1cae0a5f902-kube-api-access-42wm9\") pod \"tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw\" (UID: \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"
Jul 20 10:51:09.132069 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:09.132072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8f4a9891-73f7-4412-97a6-d1cae0a5f902-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw\" (UID: \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"
Jul 20 10:51:09.132326 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:09.132100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f4a9891-73f7-4412-97a6-d1cae0a5f902-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw\" (UID: \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"
Jul 20 10:51:09.132326 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:09.132122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f4a9891-73f7-4412-97a6-d1cae0a5f902-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw\" (UID: \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"
Jul 20 10:51:09.132480 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:09.132460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f4a9891-73f7-4412-97a6-d1cae0a5f902-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw\" (UID: \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"
Jul 20 10:51:09.132554 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:09.132537    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8f4a9891-73f7-4412-97a6-d1cae0a5f902-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw\" (UID: \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"
Jul 20 10:51:09.134682 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:09.134659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f4a9891-73f7-4412-97a6-d1cae0a5f902-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw\" (UID: \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"
Jul 20 10:51:09.139926 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:09.139905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42wm9\" (UniqueName: \"kubernetes.io/projected/8f4a9891-73f7-4412-97a6-d1cae0a5f902-kube-api-access-42wm9\") pod \"tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw\" (UID: \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"
Jul 20 10:51:09.255213 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:09.255184    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"
Jul 20 10:51:09.392502 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:09.392475    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"]
Jul 20 10:51:09.394662 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:51:09.394636    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4a9891_73f7_4412_97a6_d1cae0a5f902.slice/crio-c610cd57c9c7a6523742dd1cbb6e811851857d9034cbd16aade4d2769e2a73e0 WatchSource:0}: Error finding container c610cd57c9c7a6523742dd1cbb6e811851857d9034cbd16aade4d2769e2a73e0: Status 404 returned error can't find the container with id c610cd57c9c7a6523742dd1cbb6e811851857d9034cbd16aade4d2769e2a73e0
Jul 20 10:51:09.475107 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:09.475052    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw" event={"ID":"8f4a9891-73f7-4412-97a6-d1cae0a5f902","Type":"ContainerStarted","Data":"8aa300a29c192657a0115d7de19a9375fb5914ff49e4a339f83de96208d78f82"}
Jul 20 10:51:09.475323 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:09.475125    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw" event={"ID":"8f4a9891-73f7-4412-97a6-d1cae0a5f902","Type":"ContainerStarted","Data":"c610cd57c9c7a6523742dd1cbb6e811851857d9034cbd16aade4d2769e2a73e0"}
Jul 20 10:51:10.482277 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:10.482242    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f4a9891-73f7-4412-97a6-d1cae0a5f902" containerID="8aa300a29c192657a0115d7de19a9375fb5914ff49e4a339f83de96208d78f82" exitCode=0
Jul 20 10:51:10.482674 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:10.482314    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw" event={"ID":"8f4a9891-73f7-4412-97a6-d1cae0a5f902","Type":"ContainerDied","Data":"8aa300a29c192657a0115d7de19a9375fb5914ff49e4a339f83de96208d78f82"}
Jul 20 10:51:11.489474 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:11.489439    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw" event={"ID":"8f4a9891-73f7-4412-97a6-d1cae0a5f902","Type":"ContainerStarted","Data":"f384e0ad57da1a7cbe96596de59a28a6c30d55b411c175842470b55c75fdcb12"}
Jul 20 10:51:11.489968 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:11.489566    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"
Jul 20 10:51:11.509227 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:11.509139    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw" podStartSLOduration=2.854187246 podStartE2EDuration="3.509125129s" totalImagesPullingTime="0s" totalInitContainerRuntime="654.937883ms" isStatefulPod=false podCreationTimestamp="2026-07-20 10:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:51:11.507548723 +0000 UTC m=+3283.556412382" watchObservedRunningTime="2026-07-20 10:51:11.509125129 +0000 UTC m=+3283.557988775"
Jul 20 10:51:28.421118 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:28.421046    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:51:42.495483 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:51:42.495452    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"
Jul 20 10:52:56.933281 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:52:56.933240    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:53:00.778235 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:00.778201    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:53:00.832980 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:00.832947    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:53:14.069105 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:14.069060    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"]
Jul 20 10:53:14.078994 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:14.078965    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:53:14.138491 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:14.138460    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:53:22.495838 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:22.495798    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw" podUID="8f4a9891-73f7-4412-97a6-d1cae0a5f902" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:53:29.098001 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:29.097935    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw" podUID="8f4a9891-73f7-4412-97a6-d1cae0a5f902" containerName="main" containerID="cri-o://f384e0ad57da1a7cbe96596de59a28a6c30d55b411c175842470b55c75fdcb12" gracePeriod=45
Jul 20 10:53:29.343681 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:29.343657    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"
Jul 20 10:53:29.420105 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:29.420025    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f4a9891-73f7-4412-97a6-d1cae0a5f902-tls-certs\") pod \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\" (UID: \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\") "
Jul 20 10:53:29.420105 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:29.420098    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8f4a9891-73f7-4412-97a6-d1cae0a5f902-tokenizer-uds\") pod \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\" (UID: \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\") "
Jul 20 10:53:29.420346 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:29.420118    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42wm9\" (UniqueName: \"kubernetes.io/projected/8f4a9891-73f7-4412-97a6-d1cae0a5f902-kube-api-access-42wm9\") pod \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\" (UID: \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\") "
Jul 20 10:53:29.420346 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:29.420146    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f4a9891-73f7-4412-97a6-d1cae0a5f902-kserve-provision-location\") pod \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\" (UID: \"8f4a9891-73f7-4412-97a6-d1cae0a5f902\") "
Jul 20 10:53:29.420463 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:29.420385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f4a9891-73f7-4412-97a6-d1cae0a5f902-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8f4a9891-73f7-4412-97a6-d1cae0a5f902" (UID: "8f4a9891-73f7-4412-97a6-d1cae0a5f902"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:53:29.420904 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:29.420877    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f4a9891-73f7-4412-97a6-d1cae0a5f902-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f4a9891-73f7-4412-97a6-d1cae0a5f902" (UID: "8f4a9891-73f7-4412-97a6-d1cae0a5f902"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:53:29.422506 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:29.422478    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f4a9891-73f7-4412-97a6-d1cae0a5f902-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f4a9891-73f7-4412-97a6-d1cae0a5f902" (UID: "8f4a9891-73f7-4412-97a6-d1cae0a5f902"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:53:29.422606 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:29.422516    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4a9891-73f7-4412-97a6-d1cae0a5f902-kube-api-access-42wm9" (OuterVolumeSpecName: "kube-api-access-42wm9") pod "8f4a9891-73f7-4412-97a6-d1cae0a5f902" (UID: "8f4a9891-73f7-4412-97a6-d1cae0a5f902"). InnerVolumeSpecName "kube-api-access-42wm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:53:29.520717 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:29.520682    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f4a9891-73f7-4412-97a6-d1cae0a5f902-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:53:29.520717 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:29.520711    2580 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8f4a9891-73f7-4412-97a6-d1cae0a5f902-tokenizer-uds\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:53:29.520717 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:29.520721    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42wm9\" (UniqueName: \"kubernetes.io/projected/8f4a9891-73f7-4412-97a6-d1cae0a5f902-kube-api-access-42wm9\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:53:29.520957 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:29.520731    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f4a9891-73f7-4412-97a6-d1cae0a5f902-kserve-provision-location\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:53:30.193615 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:30.193583    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f4a9891-73f7-4412-97a6-d1cae0a5f902" containerID="f384e0ad57da1a7cbe96596de59a28a6c30d55b411c175842470b55c75fdcb12" exitCode=0
Jul 20 10:53:30.194116 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:30.193616    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw" event={"ID":"8f4a9891-73f7-4412-97a6-d1cae0a5f902","Type":"ContainerDied","Data":"f384e0ad57da1a7cbe96596de59a28a6c30d55b411c175842470b55c75fdcb12"}
Jul 20 10:53:30.194116 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:30.193647    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"
Jul 20 10:53:30.194116 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:30.193672    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw" event={"ID":"8f4a9891-73f7-4412-97a6-d1cae0a5f902","Type":"ContainerDied","Data":"c610cd57c9c7a6523742dd1cbb6e811851857d9034cbd16aade4d2769e2a73e0"}
Jul 20 10:53:30.194116 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:30.193695    2580 scope.go:117] "RemoveContainer" containerID="f384e0ad57da1a7cbe96596de59a28a6c30d55b411c175842470b55c75fdcb12"
Jul 20 10:53:30.203399 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:30.203379    2580 scope.go:117] "RemoveContainer" containerID="8aa300a29c192657a0115d7de19a9375fb5914ff49e4a339f83de96208d78f82"
Jul 20 10:53:30.211381 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:30.211365    2580 scope.go:117] "RemoveContainer" containerID="f384e0ad57da1a7cbe96596de59a28a6c30d55b411c175842470b55c75fdcb12"
Jul 20 10:53:30.211660 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:53:30.211642    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f384e0ad57da1a7cbe96596de59a28a6c30d55b411c175842470b55c75fdcb12\": container with ID starting with f384e0ad57da1a7cbe96596de59a28a6c30d55b411c175842470b55c75fdcb12 not found: ID does not exist" containerID="f384e0ad57da1a7cbe96596de59a28a6c30d55b411c175842470b55c75fdcb12"
Jul 20 10:53:30.211712 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:30.211672    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f384e0ad57da1a7cbe96596de59a28a6c30d55b411c175842470b55c75fdcb12"} err="failed to get container status \"f384e0ad57da1a7cbe96596de59a28a6c30d55b411c175842470b55c75fdcb12\": rpc error: code = NotFound desc = could not find container \"f384e0ad57da1a7cbe96596de59a28a6c30d55b411c175842470b55c75fdcb12\": container with ID starting with f384e0ad57da1a7cbe96596de59a28a6c30d55b411c175842470b55c75fdcb12 not found: ID does not exist"
Jul 20 10:53:30.211712 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:30.211691    2580 scope.go:117] "RemoveContainer" containerID="8aa300a29c192657a0115d7de19a9375fb5914ff49e4a339f83de96208d78f82"
Jul 20 10:53:30.211913 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:53:30.211894    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa300a29c192657a0115d7de19a9375fb5914ff49e4a339f83de96208d78f82\": container with ID starting with 8aa300a29c192657a0115d7de19a9375fb5914ff49e4a339f83de96208d78f82 not found: ID does not exist" containerID="8aa300a29c192657a0115d7de19a9375fb5914ff49e4a339f83de96208d78f82"
Jul 20 10:53:30.211962 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:30.211920    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa300a29c192657a0115d7de19a9375fb5914ff49e4a339f83de96208d78f82"} err="failed to get container status \"8aa300a29c192657a0115d7de19a9375fb5914ff49e4a339f83de96208d78f82\": rpc error: code = NotFound desc = could not find container \"8aa300a29c192657a0115d7de19a9375fb5914ff49e4a339f83de96208d78f82\": container with ID starting with 8aa300a29c192657a0115d7de19a9375fb5914ff49e4a339f83de96208d78f82 not found: ID does not exist"
Jul 20 10:53:30.216947 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:30.216927    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"]
Jul 20 10:53:30.220694 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:30.220670    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5f9855cd7c-cchlw"]
Jul 20 10:53:30.411581 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:30.411550    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f4a9891-73f7-4412-97a6-d1cae0a5f902" path="/var/lib/kubelet/pods/8f4a9891-73f7-4412-97a6-d1cae0a5f902/volumes"
Jul 20 10:53:34.096303 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:34.096269    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:53:34.194855 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:34.194819    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:53:39.434426 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.434385    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:53:39.732722 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.732690    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"]
Jul 20 10:53:39.733102 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.733083    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f4a9891-73f7-4412-97a6-d1cae0a5f902" containerName="main"
Jul 20 10:53:39.733223 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.733105    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f4a9891-73f7-4412-97a6-d1cae0a5f902" containerName="main"
Jul 20 10:53:39.733223 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.733133    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f4a9891-73f7-4412-97a6-d1cae0a5f902" containerName="storage-initializer"
Jul 20 10:53:39.733223 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.733142    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f4a9891-73f7-4412-97a6-d1cae0a5f902" containerName="storage-initializer"
Jul 20 10:53:39.733398 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.733262    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f4a9891-73f7-4412-97a6-d1cae0a5f902" containerName="main"
Jul 20 10:53:39.736358 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.736338    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"
Jul 20 10:53:39.738698 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.738671    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-t9s4k\""
Jul 20 10:53:39.738802 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.738674    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 20 10:53:39.739321 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.739294    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Jul 20 10:53:39.739421 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.739407    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 20 10:53:39.739496 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.739482    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 20 10:53:39.739576 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.739561    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-5nj65\""
Jul 20 10:53:39.745439 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.745419    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"]
Jul 20 10:53:39.759137 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.759112    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:53:39.808416 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.808386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/29e19f50-0db6-478a-b361-728da014f51b-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6\" (UID: \"29e19f50-0db6-478a-b361-728da014f51b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"
Jul 20 10:53:39.808610 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.808431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e19f50-0db6-478a-b361-728da014f51b-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6\" (UID: \"29e19f50-0db6-478a-b361-728da014f51b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"
Jul 20 10:53:39.808610 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.808507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e19f50-0db6-478a-b361-728da014f51b-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6\" (UID: \"29e19f50-0db6-478a-b361-728da014f51b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"
Jul 20 10:53:39.808610 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.808574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87ddp\" (UniqueName: \"kubernetes.io/projected/29e19f50-0db6-478a-b361-728da014f51b-kube-api-access-87ddp\") pod \"prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6\" (UID: \"29e19f50-0db6-478a-b361-728da014f51b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"
Jul 20 10:53:39.909361 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.909319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/29e19f50-0db6-478a-b361-728da014f51b-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6\" (UID: \"29e19f50-0db6-478a-b361-728da014f51b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"
Jul 20 10:53:39.909582 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.909376    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e19f50-0db6-478a-b361-728da014f51b-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6\" (UID: \"29e19f50-0db6-478a-b361-728da014f51b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"
Jul 20 10:53:39.909582 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.909419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e19f50-0db6-478a-b361-728da014f51b-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6\" (UID: \"29e19f50-0db6-478a-b361-728da014f51b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"
Jul 20 10:53:39.909582 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.909464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87ddp\" (UniqueName: \"kubernetes.io/projected/29e19f50-0db6-478a-b361-728da014f51b-kube-api-access-87ddp\") pod \"prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6\" (UID: \"29e19f50-0db6-478a-b361-728da014f51b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"
Jul 20 10:53:39.909807 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.909778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/29e19f50-0db6-478a-b361-728da014f51b-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6\" (UID: \"29e19f50-0db6-478a-b361-728da014f51b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"
Jul 20 10:53:39.909864 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.909791    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e19f50-0db6-478a-b361-728da014f51b-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6\" (UID: \"29e19f50-0db6-478a-b361-728da014f51b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"
Jul 20 10:53:39.911976 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.911958    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e19f50-0db6-478a-b361-728da014f51b-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6\" (UID: \"29e19f50-0db6-478a-b361-728da014f51b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"
Jul 20 10:53:39.916868 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:39.916842    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87ddp\" (UniqueName: \"kubernetes.io/projected/29e19f50-0db6-478a-b361-728da014f51b-kube-api-access-87ddp\") pod \"prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6\" (UID: \"29e19f50-0db6-478a-b361-728da014f51b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"
Jul 20 10:53:40.048442 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:40.048342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"
Jul 20 10:53:40.184866 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:40.184843    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"]
Jul 20 10:53:40.187488 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:53:40.187458    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e19f50_0db6_478a_b361_728da014f51b.slice/crio-b457d1f77d187b0b7b0822c18b994d221b9ba480995a9a7348b420d7ea49302a WatchSource:0}: Error finding container b457d1f77d187b0b7b0822c18b994d221b9ba480995a9a7348b420d7ea49302a: Status 404 returned error can't find the container with id b457d1f77d187b0b7b0822c18b994d221b9ba480995a9a7348b420d7ea49302a
Jul 20 10:53:40.245079 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:40.245050    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6" event={"ID":"29e19f50-0db6-478a-b361-728da014f51b","Type":"ContainerStarted","Data":"b457d1f77d187b0b7b0822c18b994d221b9ba480995a9a7348b420d7ea49302a"}
Jul 20 10:53:41.251872 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:41.251836    2580 generic.go:356] "Generic (PLEG): container finished" podID="29e19f50-0db6-478a-b361-728da014f51b" containerID="f4275ab8b3ce850600fa8e0d0a958b3f0018c2fcde3ab9d675126afc77088a81" exitCode=0
Jul 20 10:53:41.252301 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:41.251884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6" event={"ID":"29e19f50-0db6-478a-b361-728da014f51b","Type":"ContainerDied","Data":"f4275ab8b3ce850600fa8e0d0a958b3f0018c2fcde3ab9d675126afc77088a81"}
Jul 20 10:53:41.673881 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:41.673794    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:53:42.258446 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:42.258408    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6" event={"ID":"29e19f50-0db6-478a-b361-728da014f51b","Type":"ContainerStarted","Data":"0d6b48ff89d14f9ecc49b77b8d05940190ff993c1ea62923d8e7d6917169bedd"}
Jul 20 10:53:42.258856 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:42.258504    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"
Jul 20 10:53:42.277347 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:42.277299    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6" podStartSLOduration=2.650063207 podStartE2EDuration="3.277284108s" totalImagesPullingTime="0s" totalInitContainerRuntime="627.220901ms" isStatefulPod=false podCreationTimestamp="2026-07-20 10:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:53:42.274218292 +0000 UTC m=+3434.323081940" watchObservedRunningTime="2026-07-20 10:53:42.277284108 +0000 UTC m=+3434.326147754"
Jul 20 10:53:47.306350 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:47.306298    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:53:47.406182 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:47.406124    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:53:57.033474 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:53:57.033440    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:54:13.264987 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:13.264957    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"
Jul 20 10:54:22.923664 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:22.923628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:54:23.021331 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:23.021293    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:54:23.088214 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:23.088115    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:54:28.133142 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:28.133098    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:54:28.233227 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:28.233159    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:54:33.023070 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.023031    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:54:33.124072 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.124035    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:54:33.224944 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.224909    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:54:33.229639 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.229609    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 20 10:54:33.234400 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.234374    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.238037 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.237990    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-mn-dockercfg-8rgxn\""
Jul 20 10:54:33.238220 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.237990    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"default-dockercfg-vpm2w\""
Jul 20 10:54:33.238294 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.238223    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"openshift-service-ca.crt\""
Jul 20 10:54:33.238357 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.238324    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"kube-root-ca.crt\""
Jul 20 10:54:33.238357 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.238327    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-self-signed-certs\""
Jul 20 10:54:33.259444 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.259410    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 20 10:54:33.260237 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.260209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.260447 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.260431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.260647 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.260633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0531da77-6164-480e-b4f4-159f53da5898-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.260808 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.260792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.260938 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.260925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dftm\" (UniqueName: \"kubernetes.io/projected/0531da77-6164-480e-b4f4-159f53da5898-kube-api-access-7dftm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.261053 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.261041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.362298 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.362156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0531da77-6164-480e-b4f4-159f53da5898-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.362298 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.362263    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.362298 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.362294    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dftm\" (UniqueName: \"kubernetes.io/projected/0531da77-6164-480e-b4f4-159f53da5898-kube-api-access-7dftm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.362575 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.362312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.362575 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.362399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.362575 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.362451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.362764 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.362746    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.362826 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.362801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.362870 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.362847    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.364690 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.364666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.364971 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.364953    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0531da77-6164-480e-b4f4-159f53da5898-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.372015 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.371991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dftm\" (UniqueName: \"kubernetes.io/projected/0531da77-6164-480e-b4f4-159f53da5898-kube-api-access-7dftm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.544918 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.544871    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 10:54:33.688924 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:33.688858    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 20 10:54:33.691442 ip-10-0-133-160 kubenswrapper[2580]: W0720 10:54:33.691417    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0531da77_6164_480e_b4f4_159f53da5898.slice/crio-239432231d93c2660e12038361ef2783bba3089ac3d08c0b77ecb6ee8a8a66c4 WatchSource:0}: Error finding container 239432231d93c2660e12038361ef2783bba3089ac3d08c0b77ecb6ee8a8a66c4: Status 404 returned error can't find the container with id 239432231d93c2660e12038361ef2783bba3089ac3d08c0b77ecb6ee8a8a66c4
Jul 20 10:54:34.526486 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:34.526455    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"0531da77-6164-480e-b4f4-159f53da5898","Type":"ContainerStarted","Data":"313aa8c1d15771b0d0756b4995a1514c298e67fd0027093f9ca714dcc45582bf"}
Jul 20 10:54:34.526905 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:34.526509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"0531da77-6164-480e-b4f4-159f53da5898","Type":"ContainerStarted","Data":"239432231d93c2660e12038361ef2783bba3089ac3d08c0b77ecb6ee8a8a66c4"}
Jul 20 10:54:34.544704 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:34.544645    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podStartSLOduration=1.544627773 podStartE2EDuration="1.544627773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:54:34.542995382 +0000 UTC m=+3486.591859030" watchObservedRunningTime="2026-07-20 10:54:34.544627773 +0000 UTC m=+3486.593491500"
Jul 20 10:54:34.829273 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:34.829155    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:54:40.736397 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:40.736349    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:54:40.835771 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:54:40.835727    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:55:05.351139 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:55:05.351046    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:55:58.303080 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:55:58.303009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:55:58.306364 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:55:58.306342    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"]
Jul 20 10:56:01.958353 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:01.958308    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:56:03.265588 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:03.265551    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6" podUID="29e19f50-0db6-478a-b361-728da014f51b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:56:13.265314 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:13.265273    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6" podUID="29e19f50-0db6-478a-b361-728da014f51b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:56:13.327184 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:13.327132    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6" podUID="29e19f50-0db6-478a-b361-728da014f51b" containerName="main" containerID="cri-o://0d6b48ff89d14f9ecc49b77b8d05940190ff993c1ea62923d8e7d6917169bedd" gracePeriod=45
Jul 20 10:56:13.578353 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:13.578321    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"
Jul 20 10:56:13.637564 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:13.637525    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87ddp\" (UniqueName: \"kubernetes.io/projected/29e19f50-0db6-478a-b361-728da014f51b-kube-api-access-87ddp\") pod \"29e19f50-0db6-478a-b361-728da014f51b\" (UID: \"29e19f50-0db6-478a-b361-728da014f51b\") "
Jul 20 10:56:13.637564 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:13.637564    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/29e19f50-0db6-478a-b361-728da014f51b-tokenizer-uds\") pod \"29e19f50-0db6-478a-b361-728da014f51b\" (UID: \"29e19f50-0db6-478a-b361-728da014f51b\") "
Jul 20 10:56:13.637772 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:13.637602    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e19f50-0db6-478a-b361-728da014f51b-kserve-provision-location\") pod \"29e19f50-0db6-478a-b361-728da014f51b\" (UID: \"29e19f50-0db6-478a-b361-728da014f51b\") "
Jul 20 10:56:13.637772 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:13.637663    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e19f50-0db6-478a-b361-728da014f51b-tls-certs\") pod \"29e19f50-0db6-478a-b361-728da014f51b\" (UID: \"29e19f50-0db6-478a-b361-728da014f51b\") "
Jul 20 10:56:13.637889 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:13.637856    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e19f50-0db6-478a-b361-728da014f51b-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "29e19f50-0db6-478a-b361-728da014f51b" (UID: "29e19f50-0db6-478a-b361-728da014f51b"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:56:13.637950 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:13.637935    2580 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/29e19f50-0db6-478a-b361-728da014f51b-tokenizer-uds\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:56:13.638338 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:13.638316    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e19f50-0db6-478a-b361-728da014f51b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "29e19f50-0db6-478a-b361-728da014f51b" (UID: "29e19f50-0db6-478a-b361-728da014f51b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:56:13.640039 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:13.640008    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e19f50-0db6-478a-b361-728da014f51b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29e19f50-0db6-478a-b361-728da014f51b" (UID: "29e19f50-0db6-478a-b361-728da014f51b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:56:13.640039 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:13.640012    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e19f50-0db6-478a-b361-728da014f51b-kube-api-access-87ddp" (OuterVolumeSpecName: "kube-api-access-87ddp") pod "29e19f50-0db6-478a-b361-728da014f51b" (UID: "29e19f50-0db6-478a-b361-728da014f51b"). InnerVolumeSpecName "kube-api-access-87ddp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:56:13.738411 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:13.738378    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e19f50-0db6-478a-b361-728da014f51b-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:56:13.738411 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:13.738410    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87ddp\" (UniqueName: \"kubernetes.io/projected/29e19f50-0db6-478a-b361-728da014f51b-kube-api-access-87ddp\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:56:13.738411 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:13.738420    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e19f50-0db6-478a-b361-728da014f51b-kserve-provision-location\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 10:56:14.050240 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:14.050195    2580 generic.go:356] "Generic (PLEG): container finished" podID="29e19f50-0db6-478a-b361-728da014f51b" containerID="0d6b48ff89d14f9ecc49b77b8d05940190ff993c1ea62923d8e7d6917169bedd" exitCode=0
Jul 20 10:56:14.050446 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:14.050251    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"
Jul 20 10:56:14.050446 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:14.050265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6" event={"ID":"29e19f50-0db6-478a-b361-728da014f51b","Type":"ContainerDied","Data":"0d6b48ff89d14f9ecc49b77b8d05940190ff993c1ea62923d8e7d6917169bedd"}
Jul 20 10:56:14.050446 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:14.050323    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6" event={"ID":"29e19f50-0db6-478a-b361-728da014f51b","Type":"ContainerDied","Data":"b457d1f77d187b0b7b0822c18b994d221b9ba480995a9a7348b420d7ea49302a"}
Jul 20 10:56:14.050446 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:14.050347    2580 scope.go:117] "RemoveContainer" containerID="0d6b48ff89d14f9ecc49b77b8d05940190ff993c1ea62923d8e7d6917169bedd"
Jul 20 10:56:14.060256 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:14.060238    2580 scope.go:117] "RemoveContainer" containerID="f4275ab8b3ce850600fa8e0d0a958b3f0018c2fcde3ab9d675126afc77088a81"
Jul 20 10:56:14.068869 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:14.068851    2580 scope.go:117] "RemoveContainer" containerID="0d6b48ff89d14f9ecc49b77b8d05940190ff993c1ea62923d8e7d6917169bedd"
Jul 20 10:56:14.069130 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:56:14.069099    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d6b48ff89d14f9ecc49b77b8d05940190ff993c1ea62923d8e7d6917169bedd\": container with ID starting with 0d6b48ff89d14f9ecc49b77b8d05940190ff993c1ea62923d8e7d6917169bedd not found: ID does not exist" containerID="0d6b48ff89d14f9ecc49b77b8d05940190ff993c1ea62923d8e7d6917169bedd"
Jul 20 10:56:14.069231 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:14.069140    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d6b48ff89d14f9ecc49b77b8d05940190ff993c1ea62923d8e7d6917169bedd"} err="failed to get container status \"0d6b48ff89d14f9ecc49b77b8d05940190ff993c1ea62923d8e7d6917169bedd\": rpc error: code = NotFound desc = could not find container \"0d6b48ff89d14f9ecc49b77b8d05940190ff993c1ea62923d8e7d6917169bedd\": container with ID starting with 0d6b48ff89d14f9ecc49b77b8d05940190ff993c1ea62923d8e7d6917169bedd not found: ID does not exist"
Jul 20 10:56:14.069231 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:14.069193    2580 scope.go:117] "RemoveContainer" containerID="f4275ab8b3ce850600fa8e0d0a958b3f0018c2fcde3ab9d675126afc77088a81"
Jul 20 10:56:14.069618 ip-10-0-133-160 kubenswrapper[2580]: E0720 10:56:14.069591    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4275ab8b3ce850600fa8e0d0a958b3f0018c2fcde3ab9d675126afc77088a81\": container with ID starting with f4275ab8b3ce850600fa8e0d0a958b3f0018c2fcde3ab9d675126afc77088a81 not found: ID does not exist" containerID="f4275ab8b3ce850600fa8e0d0a958b3f0018c2fcde3ab9d675126afc77088a81"
Jul 20 10:56:14.069723 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:14.069626    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4275ab8b3ce850600fa8e0d0a958b3f0018c2fcde3ab9d675126afc77088a81"} err="failed to get container status \"f4275ab8b3ce850600fa8e0d0a958b3f0018c2fcde3ab9d675126afc77088a81\": rpc error: code = NotFound desc = could not find container \"f4275ab8b3ce850600fa8e0d0a958b3f0018c2fcde3ab9d675126afc77088a81\": container with ID starting with f4275ab8b3ce850600fa8e0d0a958b3f0018c2fcde3ab9d675126afc77088a81 not found: ID does not exist"
Jul 20 10:56:14.072042 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:14.072022    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"]
Jul 20 10:56:14.075216 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:14.075195    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-757ff44597-vcxz6"]
Jul 20 10:56:14.411212 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:56:14.411119    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29e19f50-0db6-478a-b361-728da014f51b" path="/var/lib/kubelet/pods/29e19f50-0db6-478a-b361-728da014f51b/volumes"
Jul 20 10:57:03.479866 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:57:03.479829    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:57:03.578725 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:57:03.578690    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:57:09.488463 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:57:09.488421    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:57:15.099848 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:57:15.099798    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:57:15.199867 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:57:15.199842    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:57:15.423750 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:57:15.423670    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:57:15.676363 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:57:15.676287    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:57:23.705790 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:57:23.705757    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:57:55.937660 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:57:55.937540    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:57:55.999708 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:57:55.999666    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:58:04.803234 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:58:04.803198    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:58:04.878634 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:58:04.878601    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:58:17.072964 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:58:17.072930    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:58:17.233610 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:58:17.233557    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:58:23.597548 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:58:23.597514    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:58:29.518017 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:58:29.517987    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:58:29.605740 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:58:29.605711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:59:24.467693 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:24.467615    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:59:24.563678 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:24.563625    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:59:24.864188 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:24.864134    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:59:24.977537 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:24.977498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:59:29.000449 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:29.000423    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:59:31.308116 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:31.308077    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:59:31.400985 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:31.400941    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:59:53.215830 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.215798    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:59:53.876412 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.876373    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"]
Jul 20 10:59:53.876723 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.876709    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e19f50-0db6-478a-b361-728da014f51b" containerName="main"
Jul 20 10:59:53.876771 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.876725    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e19f50-0db6-478a-b361-728da014f51b" containerName="main"
Jul 20 10:59:53.876771 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.876740    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e19f50-0db6-478a-b361-728da014f51b" containerName="storage-initializer"
Jul 20 10:59:53.876771 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.876745    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e19f50-0db6-478a-b361-728da014f51b" containerName="storage-initializer"
Jul 20 10:59:53.876883 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.876810    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e19f50-0db6-478a-b361-728da014f51b" containerName="main"
Jul 20 10:59:53.879934 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.879911    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"
Jul 20 10:59:53.882304 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.882282    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Jul 20 10:59:53.883024 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.883003    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Jul 20 10:59:53.883116 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.883044    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Jul 20 10:59:53.883116 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.883083    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-rgl6d\""
Jul 20 10:59:53.883116 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.883091    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 20 10:59:53.883306 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.883133    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-z5mn6\""
Jul 20 10:59:53.893210 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.893187    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"]
Jul 20 10:59:53.950553 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.950514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1f4b75e-3892-4c9e-9337-2810177a6518-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958\" (UID: \"a1f4b75e-3892-4c9e-9337-2810177a6518\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"
Jul 20 10:59:53.950716 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.950627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt8w8\" (UniqueName: \"kubernetes.io/projected/a1f4b75e-3892-4c9e-9337-2810177a6518-kube-api-access-bt8w8\") pod \"router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958\" (UID: \"a1f4b75e-3892-4c9e-9337-2810177a6518\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"
Jul 20 10:59:53.950716 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.950660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a1f4b75e-3892-4c9e-9337-2810177a6518-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958\" (UID: \"a1f4b75e-3892-4c9e-9337-2810177a6518\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"
Jul 20 10:59:53.950794 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.950753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1f4b75e-3892-4c9e-9337-2810177a6518-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958\" (UID: \"a1f4b75e-3892-4c9e-9337-2810177a6518\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"
Jul 20 10:59:53.988455 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:53.988427    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:59:54.051549 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:54.051507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1f4b75e-3892-4c9e-9337-2810177a6518-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958\" (UID: \"a1f4b75e-3892-4c9e-9337-2810177a6518\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"
Jul 20 10:59:54.051742 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:54.051558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1f4b75e-3892-4c9e-9337-2810177a6518-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958\" (UID: \"a1f4b75e-3892-4c9e-9337-2810177a6518\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"
Jul 20 10:59:54.051742 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:54.051589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt8w8\" (UniqueName: \"kubernetes.io/projected/a1f4b75e-3892-4c9e-9337-2810177a6518-kube-api-access-bt8w8\") pod \"router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958\" (UID: \"a1f4b75e-3892-4c9e-9337-2810177a6518\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"
Jul 20 10:59:54.051742 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:54.051610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a1f4b75e-3892-4c9e-9337-2810177a6518-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958\" (UID: \"a1f4b75e-3892-4c9e-9337-2810177a6518\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"
Jul 20 10:59:54.051992 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:54.051969    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1f4b75e-3892-4c9e-9337-2810177a6518-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958\" (UID: \"a1f4b75e-3892-4c9e-9337-2810177a6518\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"
Jul 20 10:59:54.052048 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:54.051970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a1f4b75e-3892-4c9e-9337-2810177a6518-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958\" (UID: \"a1f4b75e-3892-4c9e-9337-2810177a6518\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"
Jul 20 10:59:54.054228 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:54.054207    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1f4b75e-3892-4c9e-9337-2810177a6518-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958\" (UID: \"a1f4b75e-3892-4c9e-9337-2810177a6518\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"
Jul 20 10:59:54.059816 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:54.059782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt8w8\" (UniqueName: \"kubernetes.io/projected/a1f4b75e-3892-4c9e-9337-2810177a6518-kube-api-access-bt8w8\") pod \"router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958\" (UID: \"a1f4b75e-3892-4c9e-9337-2810177a6518\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"
Jul 20 10:59:54.089356 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:54.089327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 10:59:54.189942 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:54.189842    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"
Jul 20 10:59:54.329770 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:54.329745    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"]
Jul 20 10:59:55.166043 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:55.166010    2580 generic.go:356] "Generic (PLEG): container finished" podID="a1f4b75e-3892-4c9e-9337-2810177a6518" containerID="f5b9dee81afa5da88f2fcc34a594b5cbe09ca52f176a4c77aa2dba68581915b6" exitCode=0
Jul 20 10:59:55.166234 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:55.166101    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958" event={"ID":"a1f4b75e-3892-4c9e-9337-2810177a6518","Type":"ContainerDied","Data":"f5b9dee81afa5da88f2fcc34a594b5cbe09ca52f176a4c77aa2dba68581915b6"}
Jul 20 10:59:55.166234 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:55.166157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958" event={"ID":"a1f4b75e-3892-4c9e-9337-2810177a6518","Type":"ContainerStarted","Data":"18d9089a616d1aaaf9acd7837ec8d4deed424c523883a153973eacb84999f11a"}
Jul 20 10:59:56.173211 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:56.173151    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958" event={"ID":"a1f4b75e-3892-4c9e-9337-2810177a6518","Type":"ContainerStarted","Data":"38f779f660984e50aa15f571cc20dd6f7b74b9010a1ed7597b007eaf408b3dc7"}
Jul 20 10:59:56.173717 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:56.173280    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"
Jul 20 10:59:56.195514 ip-10-0-133-160 kubenswrapper[2580]: I0720 10:59:56.195459    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958" podStartSLOduration=2.571193232 podStartE2EDuration="3.195443735s" totalImagesPullingTime="0s" totalInitContainerRuntime="624.250503ms" isStatefulPod=false podCreationTimestamp="2026-07-20 10:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:59:56.192802649 +0000 UTC m=+3808.241666311" watchObservedRunningTime="2026-07-20 10:59:56.195443735 +0000 UTC m=+3808.244307393"
Jul 20 11:00:27.180915 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:00:27.180883    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"
Jul 20 11:00:42.296710 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:00:42.296674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:02:06.634225 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:02:06.634183    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:02:17.850439 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:02:17.850403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:02:19.990343 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:02:19.990265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:04:02.466233 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:02.466125    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"]
Jul 20 11:04:02.632217 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:02.632183    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:04:02.718263 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:02.718152    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:04:07.180464 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:07.180427    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958" podUID="a1f4b75e-3892-4c9e-9337-2810177a6518" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 11:04:17.180332 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:17.180298    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958" podUID="a1f4b75e-3892-4c9e-9337-2810177a6518" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 11:04:17.487027 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:17.486964    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958" podUID="a1f4b75e-3892-4c9e-9337-2810177a6518" containerName="main" containerID="cri-o://38f779f660984e50aa15f571cc20dd6f7b74b9010a1ed7597b007eaf408b3dc7" gracePeriod=45
Jul 20 11:04:17.738224 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:17.738131    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"
Jul 20 11:04:17.902682 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:17.902633    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt8w8\" (UniqueName: \"kubernetes.io/projected/a1f4b75e-3892-4c9e-9337-2810177a6518-kube-api-access-bt8w8\") pod \"a1f4b75e-3892-4c9e-9337-2810177a6518\" (UID: \"a1f4b75e-3892-4c9e-9337-2810177a6518\") "
Jul 20 11:04:17.902881 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:17.902747    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1f4b75e-3892-4c9e-9337-2810177a6518-kserve-provision-location\") pod \"a1f4b75e-3892-4c9e-9337-2810177a6518\" (UID: \"a1f4b75e-3892-4c9e-9337-2810177a6518\") "
Jul 20 11:04:17.902881 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:17.902773    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a1f4b75e-3892-4c9e-9337-2810177a6518-tokenizer-uds\") pod \"a1f4b75e-3892-4c9e-9337-2810177a6518\" (UID: \"a1f4b75e-3892-4c9e-9337-2810177a6518\") "
Jul 20 11:04:17.902881 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:17.902797    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1f4b75e-3892-4c9e-9337-2810177a6518-tls-certs\") pod \"a1f4b75e-3892-4c9e-9337-2810177a6518\" (UID: \"a1f4b75e-3892-4c9e-9337-2810177a6518\") "
Jul 20 11:04:17.903083 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:17.903026    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f4b75e-3892-4c9e-9337-2810177a6518-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "a1f4b75e-3892-4c9e-9337-2810177a6518" (UID: "a1f4b75e-3892-4c9e-9337-2810177a6518"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:04:17.903594 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:17.903557    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f4b75e-3892-4c9e-9337-2810177a6518-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a1f4b75e-3892-4c9e-9337-2810177a6518" (UID: "a1f4b75e-3892-4c9e-9337-2810177a6518"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:04:17.905063 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:17.905031    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f4b75e-3892-4c9e-9337-2810177a6518-kube-api-access-bt8w8" (OuterVolumeSpecName: "kube-api-access-bt8w8") pod "a1f4b75e-3892-4c9e-9337-2810177a6518" (UID: "a1f4b75e-3892-4c9e-9337-2810177a6518"). InnerVolumeSpecName "kube-api-access-bt8w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:04:17.905063 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:17.905036    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1f4b75e-3892-4c9e-9337-2810177a6518-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a1f4b75e-3892-4c9e-9337-2810177a6518" (UID: "a1f4b75e-3892-4c9e-9337-2810177a6518"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:04:18.003516 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:18.003421    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1f4b75e-3892-4c9e-9337-2810177a6518-kserve-provision-location\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:04:18.003516 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:18.003454    2580 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a1f4b75e-3892-4c9e-9337-2810177a6518-tokenizer-uds\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:04:18.003516 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:18.003463    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1f4b75e-3892-4c9e-9337-2810177a6518-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:04:18.003516 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:18.003472    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt8w8\" (UniqueName: \"kubernetes.io/projected/a1f4b75e-3892-4c9e-9337-2810177a6518-kube-api-access-bt8w8\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:04:18.506881 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:18.506845    2580 generic.go:356] "Generic (PLEG): container finished" podID="a1f4b75e-3892-4c9e-9337-2810177a6518" containerID="38f779f660984e50aa15f571cc20dd6f7b74b9010a1ed7597b007eaf408b3dc7" exitCode=0
Jul 20 11:04:18.507417 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:18.506904    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958" event={"ID":"a1f4b75e-3892-4c9e-9337-2810177a6518","Type":"ContainerDied","Data":"38f779f660984e50aa15f571cc20dd6f7b74b9010a1ed7597b007eaf408b3dc7"}
Jul 20 11:04:18.507417 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:18.506920    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"
Jul 20 11:04:18.507417 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:18.506943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958" event={"ID":"a1f4b75e-3892-4c9e-9337-2810177a6518","Type":"ContainerDied","Data":"18d9089a616d1aaaf9acd7837ec8d4deed424c523883a153973eacb84999f11a"}
Jul 20 11:04:18.507417 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:18.506963    2580 scope.go:117] "RemoveContainer" containerID="38f779f660984e50aa15f571cc20dd6f7b74b9010a1ed7597b007eaf408b3dc7"
Jul 20 11:04:18.515712 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:18.515692    2580 scope.go:117] "RemoveContainer" containerID="f5b9dee81afa5da88f2fcc34a594b5cbe09ca52f176a4c77aa2dba68581915b6"
Jul 20 11:04:18.523645 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:18.523623    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"]
Jul 20 11:04:18.524223 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:18.524203    2580 scope.go:117] "RemoveContainer" containerID="38f779f660984e50aa15f571cc20dd6f7b74b9010a1ed7597b007eaf408b3dc7"
Jul 20 11:04:18.524495 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:04:18.524473    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f779f660984e50aa15f571cc20dd6f7b74b9010a1ed7597b007eaf408b3dc7\": container with ID starting with 38f779f660984e50aa15f571cc20dd6f7b74b9010a1ed7597b007eaf408b3dc7 not found: ID does not exist" containerID="38f779f660984e50aa15f571cc20dd6f7b74b9010a1ed7597b007eaf408b3dc7"
Jul 20 11:04:18.524565 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:18.524503    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f779f660984e50aa15f571cc20dd6f7b74b9010a1ed7597b007eaf408b3dc7"} err="failed to get container status \"38f779f660984e50aa15f571cc20dd6f7b74b9010a1ed7597b007eaf408b3dc7\": rpc error: code = NotFound desc = could not find container \"38f779f660984e50aa15f571cc20dd6f7b74b9010a1ed7597b007eaf408b3dc7\": container with ID starting with 38f779f660984e50aa15f571cc20dd6f7b74b9010a1ed7597b007eaf408b3dc7 not found: ID does not exist"
Jul 20 11:04:18.524565 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:18.524522    2580 scope.go:117] "RemoveContainer" containerID="f5b9dee81afa5da88f2fcc34a594b5cbe09ca52f176a4c77aa2dba68581915b6"
Jul 20 11:04:18.524880 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:04:18.524853    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b9dee81afa5da88f2fcc34a594b5cbe09ca52f176a4c77aa2dba68581915b6\": container with ID starting with f5b9dee81afa5da88f2fcc34a594b5cbe09ca52f176a4c77aa2dba68581915b6 not found: ID does not exist" containerID="f5b9dee81afa5da88f2fcc34a594b5cbe09ca52f176a4c77aa2dba68581915b6"
Jul 20 11:04:18.524977 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:18.524890    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b9dee81afa5da88f2fcc34a594b5cbe09ca52f176a4c77aa2dba68581915b6"} err="failed to get container status \"f5b9dee81afa5da88f2fcc34a594b5cbe09ca52f176a4c77aa2dba68581915b6\": rpc error: code = NotFound desc = could not find container \"f5b9dee81afa5da88f2fcc34a594b5cbe09ca52f176a4c77aa2dba68581915b6\": container with ID starting with f5b9dee81afa5da88f2fcc34a594b5cbe09ca52f176a4c77aa2dba68581915b6 not found: ID does not exist"
Jul 20 11:04:18.527387 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:18.527367    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-764654fd7c-2g958"]
Jul 20 11:04:20.411480 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:20.411444    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1f4b75e-3892-4c9e-9337-2810177a6518" path="/var/lib/kubelet/pods/a1f4b75e-3892-4c9e-9337-2810177a6518/volumes"
Jul 20 11:04:22.524295 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:22.524260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:04:22.636211 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:22.636180    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:04:27.462411 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:27.462375    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:04:37.486313 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:37.486281    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:04:38.074337 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.074300    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"]
Jul 20 11:04:38.074727 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.074706    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1f4b75e-3892-4c9e-9337-2810177a6518" containerName="storage-initializer"
Jul 20 11:04:38.074820 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.074730    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1f4b75e-3892-4c9e-9337-2810177a6518" containerName="storage-initializer"
Jul 20 11:04:38.074820 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.074768    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1f4b75e-3892-4c9e-9337-2810177a6518" containerName="main"
Jul 20 11:04:38.074820 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.074776    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1f4b75e-3892-4c9e-9337-2810177a6518" containerName="main"
Jul 20 11:04:38.074986 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.074876    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1f4b75e-3892-4c9e-9337-2810177a6518" containerName="main"
Jul 20 11:04:38.079412 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.079390    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"
Jul 20 11:04:38.082436 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.082404    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-4vjfx\""
Jul 20 11:04:38.082436 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.082426    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Jul 20 11:04:38.082587 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.082404    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Jul 20 11:04:38.082587 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.082484    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 20 11:04:38.082587 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.082451    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Jul 20 11:04:38.082587 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.082404    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-npqcg\""
Jul 20 11:04:38.089777 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.089751    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"]
Jul 20 11:04:38.168577 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.168546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnrg6\" (UniqueName: \"kubernetes.io/projected/ba8a5556-bda6-4bfb-951f-78671da729f9-kube-api-access-fnrg6\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5\" (UID: \"ba8a5556-bda6-4bfb-951f-78671da729f9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"
Jul 20 11:04:38.168772 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.168581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8a5556-bda6-4bfb-951f-78671da729f9-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5\" (UID: \"ba8a5556-bda6-4bfb-951f-78671da729f9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"
Jul 20 11:04:38.168772 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.168602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba8a5556-bda6-4bfb-951f-78671da729f9-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5\" (UID: \"ba8a5556-bda6-4bfb-951f-78671da729f9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"
Jul 20 11:04:38.168772 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.168732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ba8a5556-bda6-4bfb-951f-78671da729f9-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5\" (UID: \"ba8a5556-bda6-4bfb-951f-78671da729f9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"
Jul 20 11:04:38.179159 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.179135    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:04:38.269847 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.269811    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnrg6\" (UniqueName: \"kubernetes.io/projected/ba8a5556-bda6-4bfb-951f-78671da729f9-kube-api-access-fnrg6\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5\" (UID: \"ba8a5556-bda6-4bfb-951f-78671da729f9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"
Jul 20 11:04:38.270050 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.269852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8a5556-bda6-4bfb-951f-78671da729f9-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5\" (UID: \"ba8a5556-bda6-4bfb-951f-78671da729f9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"
Jul 20 11:04:38.270050 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.269881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba8a5556-bda6-4bfb-951f-78671da729f9-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5\" (UID: \"ba8a5556-bda6-4bfb-951f-78671da729f9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"
Jul 20 11:04:38.270050 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.269934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ba8a5556-bda6-4bfb-951f-78671da729f9-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5\" (UID: \"ba8a5556-bda6-4bfb-951f-78671da729f9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"
Jul 20 11:04:38.270388 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.270368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ba8a5556-bda6-4bfb-951f-78671da729f9-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5\" (UID: \"ba8a5556-bda6-4bfb-951f-78671da729f9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"
Jul 20 11:04:38.270456 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.270389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba8a5556-bda6-4bfb-951f-78671da729f9-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5\" (UID: \"ba8a5556-bda6-4bfb-951f-78671da729f9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"
Jul 20 11:04:38.272505 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.272488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8a5556-bda6-4bfb-951f-78671da729f9-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5\" (UID: \"ba8a5556-bda6-4bfb-951f-78671da729f9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"
Jul 20 11:04:38.280303 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.280281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnrg6\" (UniqueName: \"kubernetes.io/projected/ba8a5556-bda6-4bfb-951f-78671da729f9-kube-api-access-fnrg6\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5\" (UID: \"ba8a5556-bda6-4bfb-951f-78671da729f9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"
Jul 20 11:04:38.390443 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.390350    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"
Jul 20 11:04:38.532760 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.532735    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"]
Jul 20 11:04:38.534942 ip-10-0-133-160 kubenswrapper[2580]: W0720 11:04:38.534917    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8a5556_bda6_4bfb_951f_78671da729f9.slice/crio-42b29f519158e61f9c0f6761cae5c815c9c6a983568e34eee1956b1ea7c0f407 WatchSource:0}: Error finding container 42b29f519158e61f9c0f6761cae5c815c9c6a983568e34eee1956b1ea7c0f407: Status 404 returned error can't find the container with id 42b29f519158e61f9c0f6761cae5c815c9c6a983568e34eee1956b1ea7c0f407
Jul 20 11:04:38.610937 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.610909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5" event={"ID":"ba8a5556-bda6-4bfb-951f-78671da729f9","Type":"ContainerStarted","Data":"8c0b375f90fa6d6c1ac021885a2f21ed1ab08affcff1b4df8992c34f95cb24ec"}
Jul 20 11:04:38.611066 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:38.610958    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5" event={"ID":"ba8a5556-bda6-4bfb-951f-78671da729f9","Type":"ContainerStarted","Data":"42b29f519158e61f9c0f6761cae5c815c9c6a983568e34eee1956b1ea7c0f407"}
Jul 20 11:04:39.622244 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:39.622124    2580 generic.go:356] "Generic (PLEG): container finished" podID="ba8a5556-bda6-4bfb-951f-78671da729f9" containerID="8c0b375f90fa6d6c1ac021885a2f21ed1ab08affcff1b4df8992c34f95cb24ec" exitCode=0
Jul 20 11:04:39.622808 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:39.622784    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5" event={"ID":"ba8a5556-bda6-4bfb-951f-78671da729f9","Type":"ContainerDied","Data":"8c0b375f90fa6d6c1ac021885a2f21ed1ab08affcff1b4df8992c34f95cb24ec"}
Jul 20 11:04:40.629826 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:40.629786    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5" event={"ID":"ba8a5556-bda6-4bfb-951f-78671da729f9","Type":"ContainerStarted","Data":"f743bdb7c37151716a7d46f7a3bcb69e31200f5863217e4fe56992b91aae7a84"}
Jul 20 11:04:40.630326 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:40.629922    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"
Jul 20 11:04:40.649785 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:40.649729    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5" podStartSLOduration=1.95227594 podStartE2EDuration="2.649712877s" totalImagesPullingTime="0s" totalInitContainerRuntime="697.436937ms" isStatefulPod=false podCreationTimestamp="2026-07-20 11:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:04:40.648566191 +0000 UTC m=+4092.697429838" watchObservedRunningTime="2026-07-20 11:04:40.649712877 +0000 UTC m=+4092.698576523"
Jul 20 11:04:49.529907 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:49.529869    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:04:49.668583 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:49.668554    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:04:55.260340 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:04:55.260309    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:05:05.102045 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:05:05.102013    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:05:11.635921 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:05:11.635887    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"
Jul 20 11:05:16.377613 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:05:16.377583    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:05:18.022480 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:05:18.022445    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:05:18.121099 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:05:18.121070    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:06:12.295983 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:06:12.295949    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:06:12.365085 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:06:12.365053    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:07:26.522610 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:26.522578    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:07:26.545725 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:26.545681    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"]
Jul 20 11:07:29.040783 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:29.040747    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:07:31.636283 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:31.636236    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5" podUID="ba8a5556-bda6-4bfb-951f-78671da729f9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 11:07:41.567788 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:41.567728    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5" podUID="ba8a5556-bda6-4bfb-951f-78671da729f9" containerName="main" containerID="cri-o://f743bdb7c37151716a7d46f7a3bcb69e31200f5863217e4fe56992b91aae7a84" gracePeriod=45
Jul 20 11:07:41.634801 ip-10-0-133-160 kubenswrapper[2580]: W0720 11:07:41.634776    2580 logging.go:55] [core] [Channel #881 SubChannel #882]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.55:9003", ServerName: "10.132.0.55:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.55:9003: connect: connection refused"
Jul 20 11:07:41.823219 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:41.823139    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"
Jul 20 11:07:41.937265 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:41.937236    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8a5556-bda6-4bfb-951f-78671da729f9-tls-certs\") pod \"ba8a5556-bda6-4bfb-951f-78671da729f9\" (UID: \"ba8a5556-bda6-4bfb-951f-78671da729f9\") "
Jul 20 11:07:41.937448 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:41.937283    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ba8a5556-bda6-4bfb-951f-78671da729f9-tokenizer-uds\") pod \"ba8a5556-bda6-4bfb-951f-78671da729f9\" (UID: \"ba8a5556-bda6-4bfb-951f-78671da729f9\") "
Jul 20 11:07:41.937448 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:41.937318    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba8a5556-bda6-4bfb-951f-78671da729f9-kserve-provision-location\") pod \"ba8a5556-bda6-4bfb-951f-78671da729f9\" (UID: \"ba8a5556-bda6-4bfb-951f-78671da729f9\") "
Jul 20 11:07:41.937448 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:41.937351    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnrg6\" (UniqueName: \"kubernetes.io/projected/ba8a5556-bda6-4bfb-951f-78671da729f9-kube-api-access-fnrg6\") pod \"ba8a5556-bda6-4bfb-951f-78671da729f9\" (UID: \"ba8a5556-bda6-4bfb-951f-78671da729f9\") "
Jul 20 11:07:41.937612 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:41.937584    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8a5556-bda6-4bfb-951f-78671da729f9-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "ba8a5556-bda6-4bfb-951f-78671da729f9" (UID: "ba8a5556-bda6-4bfb-951f-78671da729f9"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:07:41.938026 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:41.937999    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8a5556-bda6-4bfb-951f-78671da729f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ba8a5556-bda6-4bfb-951f-78671da729f9" (UID: "ba8a5556-bda6-4bfb-951f-78671da729f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:07:41.939635 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:41.939613    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba8a5556-bda6-4bfb-951f-78671da729f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba8a5556-bda6-4bfb-951f-78671da729f9" (UID: "ba8a5556-bda6-4bfb-951f-78671da729f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:07:41.939718 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:41.939639    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8a5556-bda6-4bfb-951f-78671da729f9-kube-api-access-fnrg6" (OuterVolumeSpecName: "kube-api-access-fnrg6") pod "ba8a5556-bda6-4bfb-951f-78671da729f9" (UID: "ba8a5556-bda6-4bfb-951f-78671da729f9"). InnerVolumeSpecName "kube-api-access-fnrg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:07:42.038634 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:42.038586    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnrg6\" (UniqueName: \"kubernetes.io/projected/ba8a5556-bda6-4bfb-951f-78671da729f9-kube-api-access-fnrg6\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:07:42.038634 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:42.038629    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8a5556-bda6-4bfb-951f-78671da729f9-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:07:42.038634 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:42.038639    2580 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ba8a5556-bda6-4bfb-951f-78671da729f9-tokenizer-uds\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:07:42.038634 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:42.038648    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba8a5556-bda6-4bfb-951f-78671da729f9-kserve-provision-location\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:07:42.571488 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:42.571454    2580 generic.go:356] "Generic (PLEG): container finished" podID="ba8a5556-bda6-4bfb-951f-78671da729f9" containerID="f743bdb7c37151716a7d46f7a3bcb69e31200f5863217e4fe56992b91aae7a84" exitCode=0
Jul 20 11:07:42.571901 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:42.571509    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"
Jul 20 11:07:42.571901 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:42.571521    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5" event={"ID":"ba8a5556-bda6-4bfb-951f-78671da729f9","Type":"ContainerDied","Data":"f743bdb7c37151716a7d46f7a3bcb69e31200f5863217e4fe56992b91aae7a84"}
Jul 20 11:07:42.571901 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:42.571585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5" event={"ID":"ba8a5556-bda6-4bfb-951f-78671da729f9","Type":"ContainerDied","Data":"42b29f519158e61f9c0f6761cae5c815c9c6a983568e34eee1956b1ea7c0f407"}
Jul 20 11:07:42.571901 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:42.571615    2580 scope.go:117] "RemoveContainer" containerID="f743bdb7c37151716a7d46f7a3bcb69e31200f5863217e4fe56992b91aae7a84"
Jul 20 11:07:42.580551 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:42.580529    2580 scope.go:117] "RemoveContainer" containerID="8c0b375f90fa6d6c1ac021885a2f21ed1ab08affcff1b4df8992c34f95cb24ec"
Jul 20 11:07:42.590024 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:42.590006    2580 scope.go:117] "RemoveContainer" containerID="f743bdb7c37151716a7d46f7a3bcb69e31200f5863217e4fe56992b91aae7a84"
Jul 20 11:07:42.590094 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:42.590020    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"]
Jul 20 11:07:42.590292 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:07:42.590272    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f743bdb7c37151716a7d46f7a3bcb69e31200f5863217e4fe56992b91aae7a84\": container with ID starting with f743bdb7c37151716a7d46f7a3bcb69e31200f5863217e4fe56992b91aae7a84 not found: ID does not exist" containerID="f743bdb7c37151716a7d46f7a3bcb69e31200f5863217e4fe56992b91aae7a84"
Jul 20 11:07:42.590377 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:42.590303    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f743bdb7c37151716a7d46f7a3bcb69e31200f5863217e4fe56992b91aae7a84"} err="failed to get container status \"f743bdb7c37151716a7d46f7a3bcb69e31200f5863217e4fe56992b91aae7a84\": rpc error: code = NotFound desc = could not find container \"f743bdb7c37151716a7d46f7a3bcb69e31200f5863217e4fe56992b91aae7a84\": container with ID starting with f743bdb7c37151716a7d46f7a3bcb69e31200f5863217e4fe56992b91aae7a84 not found: ID does not exist"
Jul 20 11:07:42.590377 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:42.590326    2580 scope.go:117] "RemoveContainer" containerID="8c0b375f90fa6d6c1ac021885a2f21ed1ab08affcff1b4df8992c34f95cb24ec"
Jul 20 11:07:42.590595 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:07:42.590578    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c0b375f90fa6d6c1ac021885a2f21ed1ab08affcff1b4df8992c34f95cb24ec\": container with ID starting with 8c0b375f90fa6d6c1ac021885a2f21ed1ab08affcff1b4df8992c34f95cb24ec not found: ID does not exist" containerID="8c0b375f90fa6d6c1ac021885a2f21ed1ab08affcff1b4df8992c34f95cb24ec"
Jul 20 11:07:42.590634 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:42.590601    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0b375f90fa6d6c1ac021885a2f21ed1ab08affcff1b4df8992c34f95cb24ec"} err="failed to get container status \"8c0b375f90fa6d6c1ac021885a2f21ed1ab08affcff1b4df8992c34f95cb24ec\": rpc error: code = NotFound desc = could not find container \"8c0b375f90fa6d6c1ac021885a2f21ed1ab08affcff1b4df8992c34f95cb24ec\": container with ID starting with 8c0b375f90fa6d6c1ac021885a2f21ed1ab08affcff1b4df8992c34f95cb24ec not found: ID does not exist"
Jul 20 11:07:42.596998 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:42.596977    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5"]
Jul 20 11:07:42.635183 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:42.635138    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche28qs5" podUID="ba8a5556-bda6-4bfb-951f-78671da729f9" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.55:9003\" within 1s: context deadline exceeded"
Jul 20 11:07:44.416821 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:44.416787    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba8a5556-bda6-4bfb-951f-78671da729f9" path="/var/lib/kubelet/pods/ba8a5556-bda6-4bfb-951f-78671da729f9/volumes"
Jul 20 11:07:46.532349 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:46.532310    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:07:46.628179 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:46.628146    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:07:51.294981 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.294948    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:07:51.507188 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.507132    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"]
Jul 20 11:07:51.507521 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.507508    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba8a5556-bda6-4bfb-951f-78671da729f9" containerName="main"
Jul 20 11:07:51.507571 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.507523    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba8a5556-bda6-4bfb-951f-78671da729f9" containerName="main"
Jul 20 11:07:51.507571 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.507536    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba8a5556-bda6-4bfb-951f-78671da729f9" containerName="storage-initializer"
Jul 20 11:07:51.507571 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.507541    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba8a5556-bda6-4bfb-951f-78671da729f9" containerName="storage-initializer"
Jul 20 11:07:51.507664 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.507599    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba8a5556-bda6-4bfb-951f-78671da729f9" containerName="main"
Jul 20 11:07:51.510592 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.510571    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"
Jul 20 11:07:51.513340 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.513304    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-bp496\""
Jul 20 11:07:51.513460 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.513338    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Jul 20 11:07:51.513460 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.513356    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-h5mmw\""
Jul 20 11:07:51.513460 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.513304    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Jul 20 11:07:51.513460 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.513310    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Jul 20 11:07:51.513460 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.513312    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 20 11:07:51.522731 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.522710    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"]
Jul 20 11:07:51.621580 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.621490    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75d2efa6-1356-4cae-bc38-dc5d9578649d-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9\" (UID: \"75d2efa6-1356-4cae-bc38-dc5d9578649d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"
Jul 20 11:07:51.621580 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.621524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d2efa6-1356-4cae-bc38-dc5d9578649d-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9\" (UID: \"75d2efa6-1356-4cae-bc38-dc5d9578649d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"
Jul 20 11:07:51.621580 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.621540    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/75d2efa6-1356-4cae-bc38-dc5d9578649d-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9\" (UID: \"75d2efa6-1356-4cae-bc38-dc5d9578649d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"
Jul 20 11:07:51.621842 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.621667    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kns9\" (UniqueName: \"kubernetes.io/projected/75d2efa6-1356-4cae-bc38-dc5d9578649d-kube-api-access-8kns9\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9\" (UID: \"75d2efa6-1356-4cae-bc38-dc5d9578649d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"
Jul 20 11:07:51.711823 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.711784    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:07:51.723052 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.723018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75d2efa6-1356-4cae-bc38-dc5d9578649d-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9\" (UID: \"75d2efa6-1356-4cae-bc38-dc5d9578649d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"
Jul 20 11:07:51.723251 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.723057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d2efa6-1356-4cae-bc38-dc5d9578649d-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9\" (UID: \"75d2efa6-1356-4cae-bc38-dc5d9578649d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"
Jul 20 11:07:51.723251 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.723085    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/75d2efa6-1356-4cae-bc38-dc5d9578649d-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9\" (UID: \"75d2efa6-1356-4cae-bc38-dc5d9578649d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"
Jul 20 11:07:51.723251 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.723147    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kns9\" (UniqueName: \"kubernetes.io/projected/75d2efa6-1356-4cae-bc38-dc5d9578649d-kube-api-access-8kns9\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9\" (UID: \"75d2efa6-1356-4cae-bc38-dc5d9578649d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"
Jul 20 11:07:51.723474 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.723453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75d2efa6-1356-4cae-bc38-dc5d9578649d-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9\" (UID: \"75d2efa6-1356-4cae-bc38-dc5d9578649d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"
Jul 20 11:07:51.723511 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.723481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/75d2efa6-1356-4cae-bc38-dc5d9578649d-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9\" (UID: \"75d2efa6-1356-4cae-bc38-dc5d9578649d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"
Jul 20 11:07:51.725735 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.725706    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d2efa6-1356-4cae-bc38-dc5d9578649d-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9\" (UID: \"75d2efa6-1356-4cae-bc38-dc5d9578649d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"
Jul 20 11:07:51.731707 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.731679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kns9\" (UniqueName: \"kubernetes.io/projected/75d2efa6-1356-4cae-bc38-dc5d9578649d-kube-api-access-8kns9\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9\" (UID: \"75d2efa6-1356-4cae-bc38-dc5d9578649d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"
Jul 20 11:07:51.821875 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.821830    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"
Jul 20 11:07:51.984877 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:51.984842    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"]
Jul 20 11:07:51.988852 ip-10-0-133-160 kubenswrapper[2580]: W0720 11:07:51.988823    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d2efa6_1356_4cae_bc38_dc5d9578649d.slice/crio-bb96a6e92c86f16b0fd4b4133a22b0a1b3480364e4324857491816a95d9825b6 WatchSource:0}: Error finding container bb96a6e92c86f16b0fd4b4133a22b0a1b3480364e4324857491816a95d9825b6: Status 404 returned error can't find the container with id bb96a6e92c86f16b0fd4b4133a22b0a1b3480364e4324857491816a95d9825b6
Jul 20 11:07:52.625916 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:52.625878    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9" event={"ID":"75d2efa6-1356-4cae-bc38-dc5d9578649d","Type":"ContainerStarted","Data":"c24bb3c328ea5b0497dd43e84b7e0440587316179b70ff27b90b02559025c63b"}
Jul 20 11:07:52.626364 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:52.625939    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9" event={"ID":"75d2efa6-1356-4cae-bc38-dc5d9578649d","Type":"ContainerStarted","Data":"bb96a6e92c86f16b0fd4b4133a22b0a1b3480364e4324857491816a95d9825b6"}
Jul 20 11:07:53.632442 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:53.632402    2580 generic.go:356] "Generic (PLEG): container finished" podID="75d2efa6-1356-4cae-bc38-dc5d9578649d" containerID="c24bb3c328ea5b0497dd43e84b7e0440587316179b70ff27b90b02559025c63b" exitCode=0
Jul 20 11:07:53.632845 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:53.632481    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9" event={"ID":"75d2efa6-1356-4cae-bc38-dc5d9578649d","Type":"ContainerDied","Data":"c24bb3c328ea5b0497dd43e84b7e0440587316179b70ff27b90b02559025c63b"}
Jul 20 11:07:53.681133 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:53.681074    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:07:54.640132 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:54.640094    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9" event={"ID":"75d2efa6-1356-4cae-bc38-dc5d9578649d","Type":"ContainerStarted","Data":"9d8a664a63b790d49765c851aaa8f33eef510cd37caaa5a4f4aca937a364b255"}
Jul 20 11:07:54.640567 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:54.640316    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"
Jul 20 11:07:54.661528 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:54.661475    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9" podStartSLOduration=2.951837269 podStartE2EDuration="3.661459016s" totalImagesPullingTime="0s" totalInitContainerRuntime="709.621747ms" isStatefulPod=false podCreationTimestamp="2026-07-20 11:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:07:54.658887546 +0000 UTC m=+4286.707751194" watchObservedRunningTime="2026-07-20 11:07:54.661459016 +0000 UTC m=+4286.710322703"
Jul 20 11:07:59.372590 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:59.372534    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:07:59.441030 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:59.440995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:07:59.610768 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:07:59.610735    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:08:25.647608 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:08:25.647521    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"
Jul 20 11:09:29.369897 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:29.369860    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:09:29.462030 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:29.461961    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:09:30.377572 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.377534    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 20 11:09:30.386647 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:09:30.386615    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 11:09:30.386647 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:09:30.386647 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 11:09:30.386647 ip-10-0-133-160 kubenswrapper[2580]:  > containerID="313aa8c1d15771b0d0756b4995a1514c298e67fd0027093f9ca714dcc45582bf" cmd=["/bin/sleep","15"]
Jul 20 11:09:30.386911 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.386661    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" message="" err=<
Jul 20 11:09:30.386911 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:09:30.386911 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 11:09:30.386911 ip-10-0-133-160 kubenswrapper[2580]:  >
Jul 20 11:09:30.386911 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:09:30.386676    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 11:09:30.386911 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:09:30.386911 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 11:09:30.386911 ip-10-0-133-160 kubenswrapper[2580]:  > pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podUID="0531da77-6164-480e-b4f4-159f53da5898" containerName="main" containerID="cri-o://313aa8c1d15771b0d0756b4995a1514c298e67fd0027093f9ca714dcc45582bf"
Jul 20 11:09:30.386911 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.386725    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podUID="0531da77-6164-480e-b4f4-159f53da5898" containerName="main" containerID="cri-o://313aa8c1d15771b0d0756b4995a1514c298e67fd0027093f9ca714dcc45582bf" gracePeriod=60
Jul 20 11:09:30.645295 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.645268    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 11:09:30.789310 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.789278    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-model-cache\") pod \"0531da77-6164-480e-b4f4-159f53da5898\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") "
Jul 20 11:09:30.789506 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.789347    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-dshm\") pod \"0531da77-6164-480e-b4f4-159f53da5898\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") "
Jul 20 11:09:30.789506 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.789410    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-home\") pod \"0531da77-6164-480e-b4f4-159f53da5898\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") "
Jul 20 11:09:30.790331 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.789857    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-model-cache" (OuterVolumeSpecName: "model-cache") pod "0531da77-6164-480e-b4f4-159f53da5898" (UID: "0531da77-6164-480e-b4f4-159f53da5898"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:09:30.790331 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.789964    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dftm\" (UniqueName: \"kubernetes.io/projected/0531da77-6164-480e-b4f4-159f53da5898-kube-api-access-7dftm\") pod \"0531da77-6164-480e-b4f4-159f53da5898\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") "
Jul 20 11:09:30.790331 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.790028    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-tmp-dir\") pod \"0531da77-6164-480e-b4f4-159f53da5898\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") "
Jul 20 11:09:30.790331 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.790064    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0531da77-6164-480e-b4f4-159f53da5898-tls-certs\") pod \"0531da77-6164-480e-b4f4-159f53da5898\" (UID: \"0531da77-6164-480e-b4f4-159f53da5898\") "
Jul 20 11:09:30.790331 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.790120    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-home" (OuterVolumeSpecName: "home") pod "0531da77-6164-480e-b4f4-159f53da5898" (UID: "0531da77-6164-480e-b4f4-159f53da5898"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:09:30.790656 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.790359    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0531da77-6164-480e-b4f4-159f53da5898" (UID: "0531da77-6164-480e-b4f4-159f53da5898"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:09:30.792383 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.791269    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-home\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:09:30.792383 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.791299    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-tmp-dir\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:09:30.792383 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.791318    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-model-cache\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:09:30.797677 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.797642    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-dshm" (OuterVolumeSpecName: "dshm") pod "0531da77-6164-480e-b4f4-159f53da5898" (UID: "0531da77-6164-480e-b4f4-159f53da5898"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:09:30.797677 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.797656    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0531da77-6164-480e-b4f4-159f53da5898-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0531da77-6164-480e-b4f4-159f53da5898" (UID: "0531da77-6164-480e-b4f4-159f53da5898"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:09:30.797838 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.797683    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0531da77-6164-480e-b4f4-159f53da5898-kube-api-access-7dftm" (OuterVolumeSpecName: "kube-api-access-7dftm") pod "0531da77-6164-480e-b4f4-159f53da5898" (UID: "0531da77-6164-480e-b4f4-159f53da5898"). InnerVolumeSpecName "kube-api-access-7dftm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:09:30.892690 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.892598    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0531da77-6164-480e-b4f4-159f53da5898-dshm\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:09:30.892690 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.892634    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dftm\" (UniqueName: \"kubernetes.io/projected/0531da77-6164-480e-b4f4-159f53da5898-kube-api-access-7dftm\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:09:30.892690 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:30.892646    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0531da77-6164-480e-b4f4-159f53da5898-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:09:31.138264 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:31.138232    2580 generic.go:356] "Generic (PLEG): container finished" podID="0531da77-6164-480e-b4f4-159f53da5898" containerID="313aa8c1d15771b0d0756b4995a1514c298e67fd0027093f9ca714dcc45582bf" exitCode=0
Jul 20 11:09:31.138450 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:31.138277    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"0531da77-6164-480e-b4f4-159f53da5898","Type":"ContainerDied","Data":"313aa8c1d15771b0d0756b4995a1514c298e67fd0027093f9ca714dcc45582bf"}
Jul 20 11:09:31.138450 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:31.138297    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 20 11:09:31.138450 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:31.138344    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"0531da77-6164-480e-b4f4-159f53da5898","Type":"ContainerDied","Data":"239432231d93c2660e12038361ef2783bba3089ac3d08c0b77ecb6ee8a8a66c4"}
Jul 20 11:09:31.138450 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:31.138365    2580 scope.go:117] "RemoveContainer" containerID="313aa8c1d15771b0d0756b4995a1514c298e67fd0027093f9ca714dcc45582bf"
Jul 20 11:09:31.148098 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:31.148082    2580 scope.go:117] "RemoveContainer" containerID="313aa8c1d15771b0d0756b4995a1514c298e67fd0027093f9ca714dcc45582bf"
Jul 20 11:09:31.148368 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:09:31.148349    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313aa8c1d15771b0d0756b4995a1514c298e67fd0027093f9ca714dcc45582bf\": container with ID starting with 313aa8c1d15771b0d0756b4995a1514c298e67fd0027093f9ca714dcc45582bf not found: ID does not exist" containerID="313aa8c1d15771b0d0756b4995a1514c298e67fd0027093f9ca714dcc45582bf"
Jul 20 11:09:31.148425 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:31.148378    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313aa8c1d15771b0d0756b4995a1514c298e67fd0027093f9ca714dcc45582bf"} err="failed to get container status \"313aa8c1d15771b0d0756b4995a1514c298e67fd0027093f9ca714dcc45582bf\": rpc error: code = NotFound desc = could not find container \"313aa8c1d15771b0d0756b4995a1514c298e67fd0027093f9ca714dcc45582bf\": container with ID starting with 313aa8c1d15771b0d0756b4995a1514c298e67fd0027093f9ca714dcc45582bf not found: ID does not exist"
Jul 20 11:09:31.159103 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:31.159074    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 20 11:09:31.164430 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:31.164404    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 20 11:09:32.411689 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:32.411657    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0531da77-6164-480e-b4f4-159f53da5898" path="/var/lib/kubelet/pods/0531da77-6164-480e-b4f4-159f53da5898/volumes"
Jul 20 11:09:34.357467 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:34.357430    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:09:34.458349 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:34.458322    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:09:39.690934 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:39.690893    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:09:39.800928 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:39.800896    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:09:39.886221 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:39.886156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:09:39.899324 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:39.899286    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 20 11:09:39.899707 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:39.899688    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0531da77-6164-480e-b4f4-159f53da5898" containerName="main"
Jul 20 11:09:39.899794 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:39.899709    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0531da77-6164-480e-b4f4-159f53da5898" containerName="main"
Jul 20 11:09:39.899794 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:39.899782    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0531da77-6164-480e-b4f4-159f53da5898" containerName="main"
Jul 20 11:09:39.902865 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:39.902842    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:39.905544 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:39.905518    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-mn-dockercfg-hkrp5\""
Jul 20 11:09:39.905683 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:39.905555    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"kube-root-ca.crt\""
Jul 20 11:09:39.905683 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:39.905622    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"default-dockercfg-tznxb\""
Jul 20 11:09:39.905812 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:39.905719    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"openshift-service-ca.crt\""
Jul 20 11:09:39.905870 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:39.905806    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-self-signed-certs\""
Jul 20 11:09:39.914395 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:39.914373    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 20 11:09:40.071599 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.071561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.071791 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.071612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.071791 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.071679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.071791 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.071776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.071896 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.071810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.071896 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.071831    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh4gc\" (UniqueName: \"kubernetes.io/projected/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-kube-api-access-xh4gc\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.172958 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.172917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.172958 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.172959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.173222 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.173027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.173222 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.173053    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.173222 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.173083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh4gc\" (UniqueName: \"kubernetes.io/projected/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-kube-api-access-xh4gc\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.173222 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.173120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.173482 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.173448    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.173587 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.173492    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.173587 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.173521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.175505 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.175473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.175669 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.175652    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.181264 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.181241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh4gc\" (UniqueName: \"kubernetes.io/projected/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-kube-api-access-xh4gc\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.214205 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.214178    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:09:40.359232 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:40.359204    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 20 11:09:40.360350 ip-10-0-133-160 kubenswrapper[2580]: W0720 11:09:40.360320    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf60339_7cf1_4dd5_8d81_6dcea1605b26.slice/crio-2d42b6a2a72e15222976cf61402ef446e5ff3e52a940368b376ba8c1a95d2c8e WatchSource:0}: Error finding container 2d42b6a2a72e15222976cf61402ef446e5ff3e52a940368b376ba8c1a95d2c8e: Status 404 returned error can't find the container with id 2d42b6a2a72e15222976cf61402ef446e5ff3e52a940368b376ba8c1a95d2c8e
Jul 20 11:09:41.195017 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:41.194976    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"dbf60339-7cf1-4dd5-8d81-6dcea1605b26","Type":"ContainerStarted","Data":"99a3947d7e94c3948a29e3f4c16b2d71bdf9f69d662e2d49a67e39a664f3a9de"}
Jul 20 11:09:41.195464 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:41.195037    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"dbf60339-7cf1-4dd5-8d81-6dcea1605b26","Type":"ContainerStarted","Data":"2d42b6a2a72e15222976cf61402ef446e5ff3e52a940368b376ba8c1a95d2c8e"}
Jul 20 11:09:41.212958 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:41.212907    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podStartSLOduration=2.212892073 podStartE2EDuration="2.212892073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:09:41.210210819 +0000 UTC m=+4393.259074460" watchObservedRunningTime="2026-07-20 11:09:41.212892073 +0000 UTC m=+4393.261755720"
Jul 20 11:09:41.255673 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:41.255645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:09:44.019906 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:44.019876    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:09:45.063057 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:45.063019    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:09:46.439130 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:46.439092    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:09:47.921949 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:47.921914    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:09:48.076157 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:48.076123    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:09:48.294935 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:48.294900    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:09:50.239058 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:09:50.238974    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:10:25.231429 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:10:25.231394    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:10:32.760439 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:10:32.760400    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:10:32.861241 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:10:32.861207    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:10:34.682697 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:10:34.682662    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:10:34.780692 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:10:34.780655    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:10:35.775075 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:10:35.775039    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:10:35.877305 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:10:35.877278    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:10:35.989561 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:10:35.989527    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:10:54.656680 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:10:54.656641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:10:59.461901 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:10:59.461870    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"]
Jul 20 11:10:59.470453 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:10:59.470420    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:10:59.568147 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:10:59.568116    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:05.647631 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:05.647592    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9" podUID="75d2efa6-1356-4cae-bc38-dc5d9578649d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 11:11:09.567527 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:09.567480    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:11.816155 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:11.816118    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-w2lrm"]
Jul 20 11:11:11.819672 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:11.819646    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm"
Jul 20 11:11:11.820044 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:11.820020    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:11.821451 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:11.821435    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-sysctl-allowlist\""
Jul 20 11:11:11.864810 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:11.864778    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76580d19-a324-408b-b96c-2011f9416419-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-w2lrm\" (UID: \"76580d19-a324-408b-b96c-2011f9416419\") " pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm"
Jul 20 11:11:11.865015 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:11.864831    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2ggw\" (UniqueName: \"kubernetes.io/projected/76580d19-a324-408b-b96c-2011f9416419-kube-api-access-n2ggw\") pod \"cni-sysctl-allowlist-ds-w2lrm\" (UID: \"76580d19-a324-408b-b96c-2011f9416419\") " pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm"
Jul 20 11:11:11.865015 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:11.864908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76580d19-a324-408b-b96c-2011f9416419-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-w2lrm\" (UID: \"76580d19-a324-408b-b96c-2011f9416419\") " pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm"
Jul 20 11:11:11.865015 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:11.864936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/76580d19-a324-408b-b96c-2011f9416419-ready\") pod \"cni-sysctl-allowlist-ds-w2lrm\" (UID: \"76580d19-a324-408b-b96c-2011f9416419\") " pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm"
Jul 20 11:11:11.965818 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:11.965778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76580d19-a324-408b-b96c-2011f9416419-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-w2lrm\" (UID: \"76580d19-a324-408b-b96c-2011f9416419\") " pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm"
Jul 20 11:11:11.965992 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:11.965831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2ggw\" (UniqueName: \"kubernetes.io/projected/76580d19-a324-408b-b96c-2011f9416419-kube-api-access-n2ggw\") pod \"cni-sysctl-allowlist-ds-w2lrm\" (UID: \"76580d19-a324-408b-b96c-2011f9416419\") " pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm"
Jul 20 11:11:11.965992 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:11.965866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76580d19-a324-408b-b96c-2011f9416419-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-w2lrm\" (UID: \"76580d19-a324-408b-b96c-2011f9416419\") " pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm"
Jul 20 11:11:11.965992 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:11.965895    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/76580d19-a324-408b-b96c-2011f9416419-ready\") pod \"cni-sysctl-allowlist-ds-w2lrm\" (UID: \"76580d19-a324-408b-b96c-2011f9416419\") " pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm"
Jul 20 11:11:11.965992 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:11.965899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76580d19-a324-408b-b96c-2011f9416419-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-w2lrm\" (UID: \"76580d19-a324-408b-b96c-2011f9416419\") " pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm"
Jul 20 11:11:11.966324 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:11.966302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/76580d19-a324-408b-b96c-2011f9416419-ready\") pod \"cni-sysctl-allowlist-ds-w2lrm\" (UID: \"76580d19-a324-408b-b96c-2011f9416419\") " pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm"
Jul 20 11:11:11.966463 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:11.966443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76580d19-a324-408b-b96c-2011f9416419-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-w2lrm\" (UID: \"76580d19-a324-408b-b96c-2011f9416419\") " pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm"
Jul 20 11:11:11.973383 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:11.973364    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2ggw\" (UniqueName: \"kubernetes.io/projected/76580d19-a324-408b-b96c-2011f9416419-kube-api-access-n2ggw\") pod \"cni-sysctl-allowlist-ds-w2lrm\" (UID: \"76580d19-a324-408b-b96c-2011f9416419\") " pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm"
Jul 20 11:11:12.128764 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:12.128655    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm"
Jul 20 11:11:12.141355 ip-10-0-133-160 kubenswrapper[2580]: W0720 11:11:12.141325    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76580d19_a324_408b_b96c_2011f9416419.slice/crio-95f4c9562be382fe8f644afca1262009fb9cea91b3ed24da797b2dc7bff29bd6 WatchSource:0}: Error finding container 95f4c9562be382fe8f644afca1262009fb9cea91b3ed24da797b2dc7bff29bd6: Status 404 returned error can't find the container with id 95f4c9562be382fe8f644afca1262009fb9cea91b3ed24da797b2dc7bff29bd6
Jul 20 11:11:12.391921 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:12.391807    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:12.656517 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:12.656425    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm" event={"ID":"76580d19-a324-408b-b96c-2011f9416419","Type":"ContainerStarted","Data":"fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0"}
Jul 20 11:11:12.656517 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:12.656485    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm" event={"ID":"76580d19-a324-408b-b96c-2011f9416419","Type":"ContainerStarted","Data":"95f4c9562be382fe8f644afca1262009fb9cea91b3ed24da797b2dc7bff29bd6"}
Jul 20 11:11:12.656720 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:12.656633    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm"
Jul 20 11:11:12.675251 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:12.675206    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm" podStartSLOduration=1.6751906509999999 podStartE2EDuration="1.675190651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 11:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:11:12.673860109 +0000 UTC m=+4484.722723756" watchObservedRunningTime="2026-07-20 11:11:12.675190651 +0000 UTC m=+4484.724054294"
Jul 20 11:11:13.680356 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:13.680318    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm"
Jul 20 11:11:14.174244 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:14.174199    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:14.483545 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:14.483506    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9" podUID="75d2efa6-1356-4cae-bc38-dc5d9578649d" containerName="main" containerID="cri-o://9d8a664a63b790d49765c851aaa8f33eef510cd37caaa5a4f4aca937a364b255" gracePeriod=45
Jul 20 11:11:14.675872 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:14.675841    2580 generic.go:356] "Generic (PLEG): container finished" podID="75d2efa6-1356-4cae-bc38-dc5d9578649d" containerID="9d8a664a63b790d49765c851aaa8f33eef510cd37caaa5a4f4aca937a364b255" exitCode=0
Jul 20 11:11:14.676044 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:14.675923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9" event={"ID":"75d2efa6-1356-4cae-bc38-dc5d9578649d","Type":"ContainerDied","Data":"9d8a664a63b790d49765c851aaa8f33eef510cd37caaa5a4f4aca937a364b255"}
Jul 20 11:11:14.744368 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:14.744300    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"
Jul 20 11:11:14.792075 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:14.792038    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/75d2efa6-1356-4cae-bc38-dc5d9578649d-tokenizer-uds\") pod \"75d2efa6-1356-4cae-bc38-dc5d9578649d\" (UID: \"75d2efa6-1356-4cae-bc38-dc5d9578649d\") "
Jul 20 11:11:14.792250 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:14.792092    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d2efa6-1356-4cae-bc38-dc5d9578649d-tls-certs\") pod \"75d2efa6-1356-4cae-bc38-dc5d9578649d\" (UID: \"75d2efa6-1356-4cae-bc38-dc5d9578649d\") "
Jul 20 11:11:14.792250 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:14.792114    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75d2efa6-1356-4cae-bc38-dc5d9578649d-kserve-provision-location\") pod \"75d2efa6-1356-4cae-bc38-dc5d9578649d\" (UID: \"75d2efa6-1356-4cae-bc38-dc5d9578649d\") "
Jul 20 11:11:14.792250 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:14.792156    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kns9\" (UniqueName: \"kubernetes.io/projected/75d2efa6-1356-4cae-bc38-dc5d9578649d-kube-api-access-8kns9\") pod \"75d2efa6-1356-4cae-bc38-dc5d9578649d\" (UID: \"75d2efa6-1356-4cae-bc38-dc5d9578649d\") "
Jul 20 11:11:14.792431 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:14.792386    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d2efa6-1356-4cae-bc38-dc5d9578649d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "75d2efa6-1356-4cae-bc38-dc5d9578649d" (UID: "75d2efa6-1356-4cae-bc38-dc5d9578649d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:11:14.792952 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:14.792874    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d2efa6-1356-4cae-bc38-dc5d9578649d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75d2efa6-1356-4cae-bc38-dc5d9578649d" (UID: "75d2efa6-1356-4cae-bc38-dc5d9578649d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:11:14.794526 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:14.794506    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d2efa6-1356-4cae-bc38-dc5d9578649d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75d2efa6-1356-4cae-bc38-dc5d9578649d" (UID: "75d2efa6-1356-4cae-bc38-dc5d9578649d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:11:14.794526 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:14.794515    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d2efa6-1356-4cae-bc38-dc5d9578649d-kube-api-access-8kns9" (OuterVolumeSpecName: "kube-api-access-8kns9") pod "75d2efa6-1356-4cae-bc38-dc5d9578649d" (UID: "75d2efa6-1356-4cae-bc38-dc5d9578649d"). InnerVolumeSpecName "kube-api-access-8kns9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:11:14.849459 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:14.849422    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-w2lrm"]
Jul 20 11:11:14.893626 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:14.893589    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d2efa6-1356-4cae-bc38-dc5d9578649d-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:11:14.893626 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:14.893621    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75d2efa6-1356-4cae-bc38-dc5d9578649d-kserve-provision-location\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:11:14.893626 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:14.893632    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kns9\" (UniqueName: \"kubernetes.io/projected/75d2efa6-1356-4cae-bc38-dc5d9578649d-kube-api-access-8kns9\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:11:14.893849 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:14.893642    2580 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/75d2efa6-1356-4cae-bc38-dc5d9578649d-tokenizer-uds\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:11:15.683009 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:15.682982    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"
Jul 20 11:11:15.683200 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:15.683009    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9" event={"ID":"75d2efa6-1356-4cae-bc38-dc5d9578649d","Type":"ContainerDied","Data":"bb96a6e92c86f16b0fd4b4133a22b0a1b3480364e4324857491816a95d9825b6"}
Jul 20 11:11:15.683200 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:15.683082    2580 scope.go:117] "RemoveContainer" containerID="9d8a664a63b790d49765c851aaa8f33eef510cd37caaa5a4f4aca937a364b255"
Jul 20 11:11:15.683315 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:15.683248    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm" podUID="76580d19-a324-408b-b96c-2011f9416419" containerName="kube-multus-additional-cni-plugins" containerID="cri-o://fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0" gracePeriod=30
Jul 20 11:11:15.692864 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:15.692846    2580 scope.go:117] "RemoveContainer" containerID="c24bb3c328ea5b0497dd43e84b7e0440587316179b70ff27b90b02559025c63b"
Jul 20 11:11:15.718959 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:15.718928    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"]
Jul 20 11:11:15.722181 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:15.722147    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-669599c5h9"]
Jul 20 11:11:16.410715 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:16.410672    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75d2efa6-1356-4cae-bc38-dc5d9578649d" path="/var/lib/kubelet/pods/75d2efa6-1356-4cae-bc38-dc5d9578649d/volumes"
Jul 20 11:11:16.934386 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:16.934349    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:19.574666 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:19.574577    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:19.773119 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:19.773063    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:19.939275 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:19.939158    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:22.856741 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:22.856704    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:22.978968 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:22.978930    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:23.076432 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:23.076405    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:23.177507 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:23.177410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:23.669343 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:11:23.669300    2580 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:23.670267 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:11:23.670232    2580 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:23.671183 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:11:23.671132    2580 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:23.671267 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:11:23.671192    2580 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm" podUID="76580d19-a324-408b-b96c-2011f9416419" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Jul 20 11:11:26.883474 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:26.883380    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:29.222510 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:29.222470    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:29.320000 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:29.319966    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:29.381605 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:29.381569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:32.510456 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:32.510414    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:32.579037 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:32.579001    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:33.669228 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:11:33.669162    2580 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:33.670214 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:11:33.670158    2580 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:33.671029 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:11:33.671001    2580 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:33.671108 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:11:33.671040    2580 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm" podUID="76580d19-a324-408b-b96c-2011f9416419" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Jul 20 11:11:43.669160 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:11:43.669116    2580 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:43.670055 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:11:43.670021    2580 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:43.670910 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:11:43.670883    2580 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:43.670986 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:11:43.670922    2580 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm" podUID="76580d19-a324-408b-b96c-2011f9416419" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Jul 20 11:11:45.393595 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.393539    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:45.756913 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.756893    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-w2lrm_76580d19-a324-408b-b96c-2011f9416419/kube-multus-additional-cni-plugins/0.log"
Jul 20 11:11:45.759021 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.759006    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm"
Jul 20 11:11:45.833646 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.833614    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-w2lrm_76580d19-a324-408b-b96c-2011f9416419/kube-multus-additional-cni-plugins/0.log"
Jul 20 11:11:45.833850 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.833655    2580 generic.go:356] "Generic (PLEG): container finished" podID="76580d19-a324-408b-b96c-2011f9416419" containerID="fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0" exitCode=137
Jul 20 11:11:45.833850 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.833694    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm" event={"ID":"76580d19-a324-408b-b96c-2011f9416419","Type":"ContainerDied","Data":"fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0"}
Jul 20 11:11:45.833850 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.833707    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm"
Jul 20 11:11:45.833850 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.833736    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-w2lrm" event={"ID":"76580d19-a324-408b-b96c-2011f9416419","Type":"ContainerDied","Data":"95f4c9562be382fe8f644afca1262009fb9cea91b3ed24da797b2dc7bff29bd6"}
Jul 20 11:11:45.833850 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.833759    2580 scope.go:117] "RemoveContainer" containerID="fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0"
Jul 20 11:11:45.842514 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.842496    2580 scope.go:117] "RemoveContainer" containerID="fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0"
Jul 20 11:11:45.842791 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:11:45.842767    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0\": container with ID starting with fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0 not found: ID does not exist" containerID="fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0"
Jul 20 11:11:45.842838 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.842803    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0"} err="failed to get container status \"fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0\": rpc error: code = NotFound desc = could not find container \"fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0\": container with ID starting with fee8faf54bd5352116c3cb5c70952729bb375289a62591d15aaa6469d60270e0 not found: ID does not exist"
Jul 20 11:11:45.856147 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.856119    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2ggw\" (UniqueName: \"kubernetes.io/projected/76580d19-a324-408b-b96c-2011f9416419-kube-api-access-n2ggw\") pod \"76580d19-a324-408b-b96c-2011f9416419\" (UID: \"76580d19-a324-408b-b96c-2011f9416419\") "
Jul 20 11:11:45.856299 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.856247    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/76580d19-a324-408b-b96c-2011f9416419-ready\") pod \"76580d19-a324-408b-b96c-2011f9416419\" (UID: \"76580d19-a324-408b-b96c-2011f9416419\") "
Jul 20 11:11:45.856299 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.856277    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76580d19-a324-408b-b96c-2011f9416419-tuning-conf-dir\") pod \"76580d19-a324-408b-b96c-2011f9416419\" (UID: \"76580d19-a324-408b-b96c-2011f9416419\") "
Jul 20 11:11:45.856299 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.856293    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76580d19-a324-408b-b96c-2011f9416419-cni-sysctl-allowlist\") pod \"76580d19-a324-408b-b96c-2011f9416419\" (UID: \"76580d19-a324-408b-b96c-2011f9416419\") "
Jul 20 11:11:45.856476 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.856357    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/76580d19-a324-408b-b96c-2011f9416419-tuning-conf-dir" (OuterVolumeSpecName: "tuning-conf-dir") pod "76580d19-a324-408b-b96c-2011f9416419" (UID: "76580d19-a324-408b-b96c-2011f9416419"). InnerVolumeSpecName "tuning-conf-dir". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Jul 20 11:11:45.856538 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.856477    2580 reconciler_common.go:299] "Volume detached for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76580d19-a324-408b-b96c-2011f9416419-tuning-conf-dir\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:11:45.856603 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.856575    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76580d19-a324-408b-b96c-2011f9416419-ready" (OuterVolumeSpecName: "ready") pod "76580d19-a324-408b-b96c-2011f9416419" (UID: "76580d19-a324-408b-b96c-2011f9416419"). InnerVolumeSpecName "ready". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:11:45.856667 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.856647    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76580d19-a324-408b-b96c-2011f9416419-cni-sysctl-allowlist" (OuterVolumeSpecName: "cni-sysctl-allowlist") pod "76580d19-a324-408b-b96c-2011f9416419" (UID: "76580d19-a324-408b-b96c-2011f9416419"). InnerVolumeSpecName "cni-sysctl-allowlist". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:11:45.858731 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.858708    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76580d19-a324-408b-b96c-2011f9416419-kube-api-access-n2ggw" (OuterVolumeSpecName: "kube-api-access-n2ggw") pod "76580d19-a324-408b-b96c-2011f9416419" (UID: "76580d19-a324-408b-b96c-2011f9416419"). InnerVolumeSpecName "kube-api-access-n2ggw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:11:45.878532 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.878496    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"]
Jul 20 11:11:45.878841 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.878828    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76580d19-a324-408b-b96c-2011f9416419" containerName="kube-multus-additional-cni-plugins"
Jul 20 11:11:45.878897 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.878842    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="76580d19-a324-408b-b96c-2011f9416419" containerName="kube-multus-additional-cni-plugins"
Jul 20 11:11:45.878897 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.878868    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75d2efa6-1356-4cae-bc38-dc5d9578649d" containerName="main"
Jul 20 11:11:45.878897 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.878874    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="75d2efa6-1356-4cae-bc38-dc5d9578649d" containerName="main"
Jul 20 11:11:45.878897 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.878883    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75d2efa6-1356-4cae-bc38-dc5d9578649d" containerName="storage-initializer"
Jul 20 11:11:45.878897 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.878888    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="75d2efa6-1356-4cae-bc38-dc5d9578649d" containerName="storage-initializer"
Jul 20 11:11:45.879064 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.878941    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="76580d19-a324-408b-b96c-2011f9416419" containerName="kube-multus-additional-cni-plugins"
Jul 20 11:11:45.879064 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.878950    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="75d2efa6-1356-4cae-bc38-dc5d9578649d" containerName="main"
Jul 20 11:11:45.883417 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.883391    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"
Jul 20 11:11:45.885738 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.885711    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-s7dcb\""
Jul 20 11:11:45.885738 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.885733    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Jul 20 11:11:45.885920 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.885713    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 20 11:11:45.885920 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.885714    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Jul 20 11:11:45.885920 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.885713    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-lnwl2\""
Jul 20 11:11:45.886194 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.886177    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Jul 20 11:11:45.893920 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.893894    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"]
Jul 20 11:11:45.928763 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.928728    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:45.957635 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.957604    2580 reconciler_common.go:299] "Volume detached for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/76580d19-a324-408b-b96c-2011f9416419-ready\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:11:45.957635 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.957628    2580 reconciler_common.go:299] "Volume detached for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76580d19-a324-408b-b96c-2011f9416419-cni-sysctl-allowlist\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:11:45.957635 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.957638    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2ggw\" (UniqueName: \"kubernetes.io/projected/76580d19-a324-408b-b96c-2011f9416419-kube-api-access-n2ggw\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:11:45.976717 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:45.976684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:11:46.058723 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.058686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bzb2\" (UniqueName: \"kubernetes.io/projected/3ecb0c6a-50e7-425b-9b40-826f5179eebe-kube-api-access-9bzb2\") pod \"router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk\" (UID: \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"
Jul 20 11:11:46.058916 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.058755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ecb0c6a-50e7-425b-9b40-826f5179eebe-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk\" (UID: \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"
Jul 20 11:11:46.058916 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.058835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ecb0c6a-50e7-425b-9b40-826f5179eebe-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk\" (UID: \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"
Jul 20 11:11:46.058916 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.058876    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3ecb0c6a-50e7-425b-9b40-826f5179eebe-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk\" (UID: \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"
Jul 20 11:11:46.159820 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.159781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ecb0c6a-50e7-425b-9b40-826f5179eebe-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk\" (UID: \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"
Jul 20 11:11:46.159820 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.159810    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-w2lrm"]
Jul 20 11:11:46.160026 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.159841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ecb0c6a-50e7-425b-9b40-826f5179eebe-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk\" (UID: \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"
Jul 20 11:11:46.160026 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.159871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3ecb0c6a-50e7-425b-9b40-826f5179eebe-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk\" (UID: \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"
Jul 20 11:11:46.160026 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.159924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bzb2\" (UniqueName: \"kubernetes.io/projected/3ecb0c6a-50e7-425b-9b40-826f5179eebe-kube-api-access-9bzb2\") pod \"router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk\" (UID: \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"
Jul 20 11:11:46.160304 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.160284    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ecb0c6a-50e7-425b-9b40-826f5179eebe-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk\" (UID: \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"
Jul 20 11:11:46.160370 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.160300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3ecb0c6a-50e7-425b-9b40-826f5179eebe-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk\" (UID: \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"
Jul 20 11:11:46.162859 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.162833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ecb0c6a-50e7-425b-9b40-826f5179eebe-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk\" (UID: \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"
Jul 20 11:11:46.164011 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.163992    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-w2lrm"]
Jul 20 11:11:46.167041 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.167014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bzb2\" (UniqueName: \"kubernetes.io/projected/3ecb0c6a-50e7-425b-9b40-826f5179eebe-kube-api-access-9bzb2\") pod \"router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk\" (UID: \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"
Jul 20 11:11:46.195115 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.195087    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"
Jul 20 11:11:46.330673 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.330649    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"]
Jul 20 11:11:46.332283 ip-10-0-133-160 kubenswrapper[2580]: W0720 11:11:46.332260    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ecb0c6a_50e7_425b_9b40_826f5179eebe.slice/crio-03a16e1219ab56d0572ad5d8e4be2203637d07b96c7401a85d76bd744c36f741 WatchSource:0}: Error finding container 03a16e1219ab56d0572ad5d8e4be2203637d07b96c7401a85d76bd744c36f741: Status 404 returned error can't find the container with id 03a16e1219ab56d0572ad5d8e4be2203637d07b96c7401a85d76bd744c36f741
Jul 20 11:11:46.412292 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.412250    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76580d19-a324-408b-b96c-2011f9416419" path="/var/lib/kubelet/pods/76580d19-a324-408b-b96c-2011f9416419/volumes"
Jul 20 11:11:46.842113 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.842068    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk" event={"ID":"3ecb0c6a-50e7-425b-9b40-826f5179eebe","Type":"ContainerStarted","Data":"deec42c257d53fdf471c4ad7986a01eb69b68efe790a8de916b355c818c67a7e"}
Jul 20 11:11:46.842348 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:46.842125    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk" event={"ID":"3ecb0c6a-50e7-425b-9b40-826f5179eebe","Type":"ContainerStarted","Data":"03a16e1219ab56d0572ad5d8e4be2203637d07b96c7401a85d76bd744c36f741"}
Jul 20 11:11:47.849245 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:47.849209    2580 generic.go:356] "Generic (PLEG): container finished" podID="3ecb0c6a-50e7-425b-9b40-826f5179eebe" containerID="deec42c257d53fdf471c4ad7986a01eb69b68efe790a8de916b355c818c67a7e" exitCode=0
Jul 20 11:11:47.849245 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:47.849249    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk" event={"ID":"3ecb0c6a-50e7-425b-9b40-826f5179eebe","Type":"ContainerDied","Data":"deec42c257d53fdf471c4ad7986a01eb69b68efe790a8de916b355c818c67a7e"}
Jul 20 11:11:48.856234 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:48.856193    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk" event={"ID":"3ecb0c6a-50e7-425b-9b40-826f5179eebe","Type":"ContainerStarted","Data":"5fb8c6d7e42d0536c3ba9b396430f4f757b4d27d52fa550d3d7afc327c77eb96"}
Jul 20 11:11:48.856652 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:48.856340    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"
Jul 20 11:11:48.877593 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:11:48.877541    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk" podStartSLOduration=3.225793595 podStartE2EDuration="3.877525164s" totalImagesPullingTime="0s" totalInitContainerRuntime="651.731569ms" isStatefulPod=false podCreationTimestamp="2026-07-20 11:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:11:48.875864833 +0000 UTC m=+4520.924728470" watchObservedRunningTime="2026-07-20 11:11:48.877525164 +0000 UTC m=+4520.926388811"
Jul 20 11:12:19.863366 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:12:19.863336    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"
Jul 20 11:12:42.133642 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:12:42.133606    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:12:46.489798 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:12:46.489766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:15:14.266228 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:14.266193    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"]
Jul 20 11:15:14.282588 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:14.282546    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:15:14.357506 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:14.357466    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:15:19.863045 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:19.863008    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk" podUID="3ecb0c6a-50e7-425b-9b40-826f5179eebe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 11:15:29.289581 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.289524    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk" podUID="3ecb0c6a-50e7-425b-9b40-826f5179eebe" containerName="main" containerID="cri-o://5fb8c6d7e42d0536c3ba9b396430f4f757b4d27d52fa550d3d7afc327c77eb96" gracePeriod=45
Jul 20 11:15:29.542438 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.542361    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"
Jul 20 11:15:29.594221 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.594193    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ecb0c6a-50e7-425b-9b40-826f5179eebe-kserve-provision-location\") pod \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\" (UID: \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\") "
Jul 20 11:15:29.594398 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.594274    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bzb2\" (UniqueName: \"kubernetes.io/projected/3ecb0c6a-50e7-425b-9b40-826f5179eebe-kube-api-access-9bzb2\") pod \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\" (UID: \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\") "
Jul 20 11:15:29.594398 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.594314    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ecb0c6a-50e7-425b-9b40-826f5179eebe-tls-certs\") pod \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\" (UID: \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\") "
Jul 20 11:15:29.594398 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.594330    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3ecb0c6a-50e7-425b-9b40-826f5179eebe-tokenizer-uds\") pod \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\" (UID: \"3ecb0c6a-50e7-425b-9b40-826f5179eebe\") "
Jul 20 11:15:29.594671 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.594641    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecb0c6a-50e7-425b-9b40-826f5179eebe-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3ecb0c6a-50e7-425b-9b40-826f5179eebe" (UID: "3ecb0c6a-50e7-425b-9b40-826f5179eebe"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:15:29.594980 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.594958    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecb0c6a-50e7-425b-9b40-826f5179eebe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3ecb0c6a-50e7-425b-9b40-826f5179eebe" (UID: "3ecb0c6a-50e7-425b-9b40-826f5179eebe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:15:29.596537 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.596510    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ecb0c6a-50e7-425b-9b40-826f5179eebe-kube-api-access-9bzb2" (OuterVolumeSpecName: "kube-api-access-9bzb2") pod "3ecb0c6a-50e7-425b-9b40-826f5179eebe" (UID: "3ecb0c6a-50e7-425b-9b40-826f5179eebe"). InnerVolumeSpecName "kube-api-access-9bzb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:15:29.596648 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.596628    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ecb0c6a-50e7-425b-9b40-826f5179eebe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ecb0c6a-50e7-425b-9b40-826f5179eebe" (UID: "3ecb0c6a-50e7-425b-9b40-826f5179eebe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:15:29.695588 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.695554    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bzb2\" (UniqueName: \"kubernetes.io/projected/3ecb0c6a-50e7-425b-9b40-826f5179eebe-kube-api-access-9bzb2\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:15:29.695588 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.695584    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ecb0c6a-50e7-425b-9b40-826f5179eebe-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:15:29.695588 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.695594    2580 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3ecb0c6a-50e7-425b-9b40-826f5179eebe-tokenizer-uds\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:15:29.695827 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.695603    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ecb0c6a-50e7-425b-9b40-826f5179eebe-kserve-provision-location\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:15:29.977277 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.977240    2580 generic.go:356] "Generic (PLEG): container finished" podID="3ecb0c6a-50e7-425b-9b40-826f5179eebe" containerID="5fb8c6d7e42d0536c3ba9b396430f4f757b4d27d52fa550d3d7afc327c77eb96" exitCode=0
Jul 20 11:15:29.977456 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.977280    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk" event={"ID":"3ecb0c6a-50e7-425b-9b40-826f5179eebe","Type":"ContainerDied","Data":"5fb8c6d7e42d0536c3ba9b396430f4f757b4d27d52fa550d3d7afc327c77eb96"}
Jul 20 11:15:29.977456 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.977302    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"
Jul 20 11:15:29.977456 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.977348    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk" event={"ID":"3ecb0c6a-50e7-425b-9b40-826f5179eebe","Type":"ContainerDied","Data":"03a16e1219ab56d0572ad5d8e4be2203637d07b96c7401a85d76bd744c36f741"}
Jul 20 11:15:29.977456 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.977378    2580 scope.go:117] "RemoveContainer" containerID="5fb8c6d7e42d0536c3ba9b396430f4f757b4d27d52fa550d3d7afc327c77eb96"
Jul 20 11:15:29.986982 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.986962    2580 scope.go:117] "RemoveContainer" containerID="deec42c257d53fdf471c4ad7986a01eb69b68efe790a8de916b355c818c67a7e"
Jul 20 11:15:29.994745 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.994723    2580 scope.go:117] "RemoveContainer" containerID="5fb8c6d7e42d0536c3ba9b396430f4f757b4d27d52fa550d3d7afc327c77eb96"
Jul 20 11:15:29.995000 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:15:29.994980    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb8c6d7e42d0536c3ba9b396430f4f757b4d27d52fa550d3d7afc327c77eb96\": container with ID starting with 5fb8c6d7e42d0536c3ba9b396430f4f757b4d27d52fa550d3d7afc327c77eb96 not found: ID does not exist" containerID="5fb8c6d7e42d0536c3ba9b396430f4f757b4d27d52fa550d3d7afc327c77eb96"
Jul 20 11:15:29.995048 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.995009    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb8c6d7e42d0536c3ba9b396430f4f757b4d27d52fa550d3d7afc327c77eb96"} err="failed to get container status \"5fb8c6d7e42d0536c3ba9b396430f4f757b4d27d52fa550d3d7afc327c77eb96\": rpc error: code = NotFound desc = could not find container \"5fb8c6d7e42d0536c3ba9b396430f4f757b4d27d52fa550d3d7afc327c77eb96\": container with ID starting with 5fb8c6d7e42d0536c3ba9b396430f4f757b4d27d52fa550d3d7afc327c77eb96 not found: ID does not exist"
Jul 20 11:15:29.995048 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.995027    2580 scope.go:117] "RemoveContainer" containerID="deec42c257d53fdf471c4ad7986a01eb69b68efe790a8de916b355c818c67a7e"
Jul 20 11:15:29.995337 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:15:29.995286    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deec42c257d53fdf471c4ad7986a01eb69b68efe790a8de916b355c818c67a7e\": container with ID starting with deec42c257d53fdf471c4ad7986a01eb69b68efe790a8de916b355c818c67a7e not found: ID does not exist" containerID="deec42c257d53fdf471c4ad7986a01eb69b68efe790a8de916b355c818c67a7e"
Jul 20 11:15:29.995337 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.995315    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deec42c257d53fdf471c4ad7986a01eb69b68efe790a8de916b355c818c67a7e"} err="failed to get container status \"deec42c257d53fdf471c4ad7986a01eb69b68efe790a8de916b355c818c67a7e\": rpc error: code = NotFound desc = could not find container \"deec42c257d53fdf471c4ad7986a01eb69b68efe790a8de916b355c818c67a7e\": container with ID starting with deec42c257d53fdf471c4ad7986a01eb69b68efe790a8de916b355c818c67a7e not found: ID does not exist"
Jul 20 11:15:29.998137 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:29.998110    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"]
Jul 20 11:15:30.001099 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:30.001074    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-689d454dblzrsk"]
Jul 20 11:15:30.411754 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:30.411651    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ecb0c6a-50e7-425b-9b40-826f5179eebe" path="/var/lib/kubelet/pods/3ecb0c6a-50e7-425b-9b40-826f5179eebe/volumes"
Jul 20 11:15:34.294014 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:34.293984    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:15:39.460048 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:39.460009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:15:45.412125 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:45.412084    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:15:52.262285 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:52.262159    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:15:53.029330 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:53.029300    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:15:58.711517 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:15:58.711486    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:16:08.724381 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:16:08.724343    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:16:13.212890 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:16:13.212854    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:16:13.271189 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:16:13.271131    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:16:19.828381 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:16:19.828332    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:16:22.357101 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:16:22.357067    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:17:50.462428 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:17:50.462385    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:18:10.282291 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:10.282251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:18:10.380659 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:10.380623    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:18:16.224381 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:16.224347    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:18:19.616293 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:19.616255    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:18:19.920017 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:19.919921    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:18:20.039272 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.039239    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:18:20.404191 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.403593    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"]
Jul 20 11:18:20.406326 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.406297    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ecb0c6a-50e7-425b-9b40-826f5179eebe" containerName="main"
Jul 20 11:18:20.406790 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.406752    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ecb0c6a-50e7-425b-9b40-826f5179eebe" containerName="main"
Jul 20 11:18:20.407053 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.407036    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ecb0c6a-50e7-425b-9b40-826f5179eebe" containerName="storage-initializer"
Jul 20 11:18:20.407216 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.407199    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ecb0c6a-50e7-425b-9b40-826f5179eebe" containerName="storage-initializer"
Jul 20 11:18:20.407582 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.407563    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ecb0c6a-50e7-425b-9b40-826f5179eebe" containerName="main"
Jul 20 11:18:20.412047 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.412023    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"
Jul 20 11:18:20.414269 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.414245    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 20 11:18:20.414269 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.414249    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 20 11:18:20.414494 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.414280    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-p5pfx\""
Jul 20 11:18:20.414712 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.414683    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 20 11:18:20.414791 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.414689    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 20 11:18:20.414854 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.414791    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-q9mxj\""
Jul 20 11:18:20.423599 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.423577    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"]
Jul 20 11:18:20.430798 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.430774    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:18:20.453644 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.453613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1474ce33-ed5a-4503-a3f9-7e02be141e3a-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4\" (UID: \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"
Jul 20 11:18:20.453809 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.453650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1474ce33-ed5a-4503-a3f9-7e02be141e3a-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4\" (UID: \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"
Jul 20 11:18:20.453809 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.453681    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng9qq\" (UniqueName: \"kubernetes.io/projected/1474ce33-ed5a-4503-a3f9-7e02be141e3a-kube-api-access-ng9qq\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4\" (UID: \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"
Jul 20 11:18:20.453926 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.453892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1474ce33-ed5a-4503-a3f9-7e02be141e3a-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4\" (UID: \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"
Jul 20 11:18:20.554760 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.554720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1474ce33-ed5a-4503-a3f9-7e02be141e3a-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4\" (UID: \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"
Jul 20 11:18:20.555003 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.554771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1474ce33-ed5a-4503-a3f9-7e02be141e3a-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4\" (UID: \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"
Jul 20 11:18:20.555003 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.554807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1474ce33-ed5a-4503-a3f9-7e02be141e3a-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4\" (UID: \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"
Jul 20 11:18:20.555003 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.554846    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng9qq\" (UniqueName: \"kubernetes.io/projected/1474ce33-ed5a-4503-a3f9-7e02be141e3a-kube-api-access-ng9qq\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4\" (UID: \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"
Jul 20 11:18:20.555247 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.555223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1474ce33-ed5a-4503-a3f9-7e02be141e3a-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4\" (UID: \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"
Jul 20 11:18:20.555334 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.555288    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1474ce33-ed5a-4503-a3f9-7e02be141e3a-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4\" (UID: \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"
Jul 20 11:18:20.557665 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.557640    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1474ce33-ed5a-4503-a3f9-7e02be141e3a-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4\" (UID: \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"
Jul 20 11:18:20.562047 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.562026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng9qq\" (UniqueName: \"kubernetes.io/projected/1474ce33-ed5a-4503-a3f9-7e02be141e3a-kube-api-access-ng9qq\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4\" (UID: \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"
Jul 20 11:18:20.724361 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.724334    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"
Jul 20 11:18:20.862010 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:20.861984    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"]
Jul 20 11:18:20.864982 ip-10-0-133-160 kubenswrapper[2580]: W0720 11:18:20.864942    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1474ce33_ed5a_4503_a3f9_7e02be141e3a.slice/crio-537a183a7f2f75e3e6e95916651c2cb55ddb4031418a571f7956db5de0a490fa WatchSource:0}: Error finding container 537a183a7f2f75e3e6e95916651c2cb55ddb4031418a571f7956db5de0a490fa: Status 404 returned error can't find the container with id 537a183a7f2f75e3e6e95916651c2cb55ddb4031418a571f7956db5de0a490fa
Jul 20 11:18:21.855763 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:21.855674    2580 generic.go:356] "Generic (PLEG): container finished" podID="1474ce33-ed5a-4503-a3f9-7e02be141e3a" containerID="ef947fae72a89dac100a69c3999619f7a05412db10c843c8c9ed810af409efce" exitCode=0
Jul 20 11:18:21.855763 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:21.855733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4" event={"ID":"1474ce33-ed5a-4503-a3f9-7e02be141e3a","Type":"ContainerDied","Data":"ef947fae72a89dac100a69c3999619f7a05412db10c843c8c9ed810af409efce"}
Jul 20 11:18:21.856239 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:21.855775    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4" event={"ID":"1474ce33-ed5a-4503-a3f9-7e02be141e3a","Type":"ContainerStarted","Data":"537a183a7f2f75e3e6e95916651c2cb55ddb4031418a571f7956db5de0a490fa"}
Jul 20 11:18:22.309427 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:22.309375    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:18:22.415308 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:22.415278    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:18:22.862869 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:22.862834    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4" event={"ID":"1474ce33-ed5a-4503-a3f9-7e02be141e3a","Type":"ContainerStarted","Data":"87b8432f9ec1b88465c0caa30602eac0e482644b56e92b8f3b030472936f1e8e"}
Jul 20 11:18:22.863309 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:22.862934    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"
Jul 20 11:18:22.880656 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:22.880600    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4" podStartSLOduration=2.227477376 podStartE2EDuration="2.880574697s" totalImagesPullingTime="0s" totalInitContainerRuntime="653.097321ms" isStatefulPod=false podCreationTimestamp="2026-07-20 11:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:18:22.88019494 +0000 UTC m=+4914.929058587" watchObservedRunningTime="2026-07-20 11:18:22.880574697 +0000 UTC m=+4914.929438343"
Jul 20 11:18:30.439520 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:30.439482    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:18:53.869805 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:18:53.869728    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"
Jul 20 11:19:57.519448 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:19:57.519407    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:20:55.390336 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:20:55.390305    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:20:55.398970 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:20:55.398943    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"]
Jul 20 11:20:56.454287 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:20:56.454248    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:21:03.869962 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:03.869921    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4" podUID="1474ce33-ed5a-4503-a3f9-7e02be141e3a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 11:21:10.419737 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.419700    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4" podUID="1474ce33-ed5a-4503-a3f9-7e02be141e3a" containerName="main" containerID="cri-o://87b8432f9ec1b88465c0caa30602eac0e482644b56e92b8f3b030472936f1e8e" gracePeriod=45
Jul 20 11:21:10.673587 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.673523    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"
Jul 20 11:21:10.716299 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.716275    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1474ce33-ed5a-4503-a3f9-7e02be141e3a-kserve-provision-location\") pod \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\" (UID: \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\") "
Jul 20 11:21:10.716477 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.716320    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1474ce33-ed5a-4503-a3f9-7e02be141e3a-tls-certs\") pod \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\" (UID: \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\") "
Jul 20 11:21:10.716477 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.716355    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1474ce33-ed5a-4503-a3f9-7e02be141e3a-tokenizer-uds\") pod \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\" (UID: \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\") "
Jul 20 11:21:10.716477 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.716381    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ng9qq\" (UniqueName: \"kubernetes.io/projected/1474ce33-ed5a-4503-a3f9-7e02be141e3a-kube-api-access-ng9qq\") pod \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\" (UID: \"1474ce33-ed5a-4503-a3f9-7e02be141e3a\") "
Jul 20 11:21:10.716662 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.716594    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1474ce33-ed5a-4503-a3f9-7e02be141e3a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1474ce33-ed5a-4503-a3f9-7e02be141e3a" (UID: "1474ce33-ed5a-4503-a3f9-7e02be141e3a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:21:10.716727 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.716709    2580 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1474ce33-ed5a-4503-a3f9-7e02be141e3a-tokenizer-uds\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:21:10.717027 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.717002    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1474ce33-ed5a-4503-a3f9-7e02be141e3a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1474ce33-ed5a-4503-a3f9-7e02be141e3a" (UID: "1474ce33-ed5a-4503-a3f9-7e02be141e3a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:21:10.718647 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.718625    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1474ce33-ed5a-4503-a3f9-7e02be141e3a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1474ce33-ed5a-4503-a3f9-7e02be141e3a" (UID: "1474ce33-ed5a-4503-a3f9-7e02be141e3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:21:10.718797 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.718778    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1474ce33-ed5a-4503-a3f9-7e02be141e3a-kube-api-access-ng9qq" (OuterVolumeSpecName: "kube-api-access-ng9qq") pod "1474ce33-ed5a-4503-a3f9-7e02be141e3a" (UID: "1474ce33-ed5a-4503-a3f9-7e02be141e3a"). InnerVolumeSpecName "kube-api-access-ng9qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:21:10.722631 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.722603    2580 generic.go:356] "Generic (PLEG): container finished" podID="1474ce33-ed5a-4503-a3f9-7e02be141e3a" containerID="87b8432f9ec1b88465c0caa30602eac0e482644b56e92b8f3b030472936f1e8e" exitCode=0
Jul 20 11:21:10.722740 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.722643    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4" event={"ID":"1474ce33-ed5a-4503-a3f9-7e02be141e3a","Type":"ContainerDied","Data":"87b8432f9ec1b88465c0caa30602eac0e482644b56e92b8f3b030472936f1e8e"}
Jul 20 11:21:10.722740 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.722651    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"
Jul 20 11:21:10.722740 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.722698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4" event={"ID":"1474ce33-ed5a-4503-a3f9-7e02be141e3a","Type":"ContainerDied","Data":"537a183a7f2f75e3e6e95916651c2cb55ddb4031418a571f7956db5de0a490fa"}
Jul 20 11:21:10.722740 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.722730    2580 scope.go:117] "RemoveContainer" containerID="87b8432f9ec1b88465c0caa30602eac0e482644b56e92b8f3b030472936f1e8e"
Jul 20 11:21:10.734836 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.734815    2580 scope.go:117] "RemoveContainer" containerID="ef947fae72a89dac100a69c3999619f7a05412db10c843c8c9ed810af409efce"
Jul 20 11:21:10.742899 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.742874    2580 scope.go:117] "RemoveContainer" containerID="87b8432f9ec1b88465c0caa30602eac0e482644b56e92b8f3b030472936f1e8e"
Jul 20 11:21:10.743215 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:21:10.743190    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b8432f9ec1b88465c0caa30602eac0e482644b56e92b8f3b030472936f1e8e\": container with ID starting with 87b8432f9ec1b88465c0caa30602eac0e482644b56e92b8f3b030472936f1e8e not found: ID does not exist" containerID="87b8432f9ec1b88465c0caa30602eac0e482644b56e92b8f3b030472936f1e8e"
Jul 20 11:21:10.743308 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.743222    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b8432f9ec1b88465c0caa30602eac0e482644b56e92b8f3b030472936f1e8e"} err="failed to get container status \"87b8432f9ec1b88465c0caa30602eac0e482644b56e92b8f3b030472936f1e8e\": rpc error: code = NotFound desc = could not find container \"87b8432f9ec1b88465c0caa30602eac0e482644b56e92b8f3b030472936f1e8e\": container with ID starting with 87b8432f9ec1b88465c0caa30602eac0e482644b56e92b8f3b030472936f1e8e not found: ID does not exist"
Jul 20 11:21:10.743308 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.743240    2580 scope.go:117] "RemoveContainer" containerID="ef947fae72a89dac100a69c3999619f7a05412db10c843c8c9ed810af409efce"
Jul 20 11:21:10.743666 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:21:10.743641    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef947fae72a89dac100a69c3999619f7a05412db10c843c8c9ed810af409efce\": container with ID starting with ef947fae72a89dac100a69c3999619f7a05412db10c843c8c9ed810af409efce not found: ID does not exist" containerID="ef947fae72a89dac100a69c3999619f7a05412db10c843c8c9ed810af409efce"
Jul 20 11:21:10.743739 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.743675    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef947fae72a89dac100a69c3999619f7a05412db10c843c8c9ed810af409efce"} err="failed to get container status \"ef947fae72a89dac100a69c3999619f7a05412db10c843c8c9ed810af409efce\": rpc error: code = NotFound desc = could not find container \"ef947fae72a89dac100a69c3999619f7a05412db10c843c8c9ed810af409efce\": container with ID starting with ef947fae72a89dac100a69c3999619f7a05412db10c843c8c9ed810af409efce not found: ID does not exist"
Jul 20 11:21:10.745124 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.745105    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"]
Jul 20 11:21:10.750074 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.750055    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegt8b4"]
Jul 20 11:21:10.817360 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.817324    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1474ce33-ed5a-4503-a3f9-7e02be141e3a-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:21:10.817360 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.817357    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng9qq\" (UniqueName: \"kubernetes.io/projected/1474ce33-ed5a-4503-a3f9-7e02be141e3a-kube-api-access-ng9qq\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:21:10.817537 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:10.817371    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1474ce33-ed5a-4503-a3f9-7e02be141e3a-kserve-provision-location\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:21:12.411395 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:12.411363    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1474ce33-ed5a-4503-a3f9-7e02be141e3a" path="/var/lib/kubelet/pods/1474ce33-ed5a-4503-a3f9-7e02be141e3a/volumes"
Jul 20 11:21:30.487751 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:30.487716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:21:30.585423 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:30.585389    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:21:36.656439 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:36.656410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:21:38.483126 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:38.483095    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:21:38.664913 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:38.664879    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:21:42.301893 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:42.301862    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:21:42.359021 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:42.358988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:21:45.968845 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:21:45.968811    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:22:12.676871 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:22:12.676819    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:22:12.757288 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:22:12.757254    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:22:32.622081 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:22:32.622048    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:22:32.717452 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:22:32.717413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:22:34.311559 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:22:34.311527    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:22:34.366814 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:22:34.366783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:22:38.770965 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:22:38.770931    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:22:44.375466 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:22:44.375425    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:22:44.436964 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:22:44.436931    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:22:52.313119 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:22:52.313061    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:22:52.632436 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:22:52.632331    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:23:03.594832 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:23:03.594794    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:23:26.336237 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:23:26.336133    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:23:26.436299 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:23:26.436260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:23:45.931694 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:23:45.931659    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:23:46.032980 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:23:46.032950    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:23:51.612325 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:23:51.612284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:23:51.896365 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:23:51.896276    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:23:52.253834 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:23:52.253787    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:23:57.927858 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:23:57.927828    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:23:58.006041 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:23:58.006006    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:24:01.658388 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:01.658351    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:24:25.616607 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:25.616566    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:24:25.677082 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:25.677047    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:24:35.730037 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:35.730000    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:24:35.791336 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:35.791298    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:24:36.812108 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:36.812070    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 20 11:24:36.821125 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:24:36.821088    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 11:24:36.821125 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:24:36.821125 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 11:24:36.821125 ip-10-0-133-160 kubenswrapper[2580]:  > containerID="99a3947d7e94c3948a29e3f4c16b2d71bdf9f69d662e2d49a67e39a664f3a9de" cmd=["/bin/sleep","15"]
Jul 20 11:24:36.821450 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:36.821138    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" message="" err=<
Jul 20 11:24:36.821450 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:24:36.821450 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 11:24:36.821450 ip-10-0-133-160 kubenswrapper[2580]:  >
Jul 20 11:24:36.821450 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:24:36.821150    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 11:24:36.821450 ip-10-0-133-160 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:24:36.821450 ip-10-0-133-160 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 20 11:24:36.821450 ip-10-0-133-160 kubenswrapper[2580]:  > pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podUID="dbf60339-7cf1-4dd5-8d81-6dcea1605b26" containerName="main" containerID="cri-o://99a3947d7e94c3948a29e3f4c16b2d71bdf9f69d662e2d49a67e39a664f3a9de"
Jul 20 11:24:36.821450 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:36.821212    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podUID="dbf60339-7cf1-4dd5-8d81-6dcea1605b26" containerName="main" containerID="cri-o://99a3947d7e94c3948a29e3f4c16b2d71bdf9f69d662e2d49a67e39a664f3a9de" gracePeriod=60
Jul 20 11:24:37.070908 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.070846    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:24:37.084625 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.084601    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-model-cache\") pod \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") "
Jul 20 11:24:37.084770 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.084655    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-tmp-dir\") pod \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") "
Jul 20 11:24:37.084770 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.084682    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-home\") pod \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") "
Jul 20 11:24:37.084770 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.084713    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-dshm\") pod \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") "
Jul 20 11:24:37.084770 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.084738    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-tls-certs\") pod \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") "
Jul 20 11:24:37.084770 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.084762    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh4gc\" (UniqueName: \"kubernetes.io/projected/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-kube-api-access-xh4gc\") pod \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\" (UID: \"dbf60339-7cf1-4dd5-8d81-6dcea1605b26\") "
Jul 20 11:24:37.085029 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.084892    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-model-cache" (OuterVolumeSpecName: "model-cache") pod "dbf60339-7cf1-4dd5-8d81-6dcea1605b26" (UID: "dbf60339-7cf1-4dd5-8d81-6dcea1605b26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:24:37.085029 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.084943    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dbf60339-7cf1-4dd5-8d81-6dcea1605b26" (UID: "dbf60339-7cf1-4dd5-8d81-6dcea1605b26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:24:37.085029 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.084997    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-model-cache\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:24:37.085292 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.085245    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-home" (OuterVolumeSpecName: "home") pod "dbf60339-7cf1-4dd5-8d81-6dcea1605b26" (UID: "dbf60339-7cf1-4dd5-8d81-6dcea1605b26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:24:37.087518 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.087480    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-dshm" (OuterVolumeSpecName: "dshm") pod "dbf60339-7cf1-4dd5-8d81-6dcea1605b26" (UID: "dbf60339-7cf1-4dd5-8d81-6dcea1605b26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:24:37.087680 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.087656    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-kube-api-access-xh4gc" (OuterVolumeSpecName: "kube-api-access-xh4gc") pod "dbf60339-7cf1-4dd5-8d81-6dcea1605b26" (UID: "dbf60339-7cf1-4dd5-8d81-6dcea1605b26"). InnerVolumeSpecName "kube-api-access-xh4gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:24:37.087795 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.087774    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dbf60339-7cf1-4dd5-8d81-6dcea1605b26" (UID: "dbf60339-7cf1-4dd5-8d81-6dcea1605b26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:24:37.185917 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.185865    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-tmp-dir\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:24:37.185917 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.185910    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-home\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:24:37.185917 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.185920    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-dshm\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:24:37.185917 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.185927    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:24:37.185917 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.185936    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh4gc\" (UniqueName: \"kubernetes.io/projected/dbf60339-7cf1-4dd5-8d81-6dcea1605b26-kube-api-access-xh4gc\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:24:37.762036 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.762003    2580 generic.go:356] "Generic (PLEG): container finished" podID="dbf60339-7cf1-4dd5-8d81-6dcea1605b26" containerID="99a3947d7e94c3948a29e3f4c16b2d71bdf9f69d662e2d49a67e39a664f3a9de" exitCode=0
Jul 20 11:24:37.762244 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.762060    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 20 11:24:37.762244 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.762094    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"dbf60339-7cf1-4dd5-8d81-6dcea1605b26","Type":"ContainerDied","Data":"99a3947d7e94c3948a29e3f4c16b2d71bdf9f69d662e2d49a67e39a664f3a9de"}
Jul 20 11:24:37.762244 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.762190    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"dbf60339-7cf1-4dd5-8d81-6dcea1605b26","Type":"ContainerDied","Data":"2d42b6a2a72e15222976cf61402ef446e5ff3e52a940368b376ba8c1a95d2c8e"}
Jul 20 11:24:37.762244 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.762217    2580 scope.go:117] "RemoveContainer" containerID="99a3947d7e94c3948a29e3f4c16b2d71bdf9f69d662e2d49a67e39a664f3a9de"
Jul 20 11:24:37.772450 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.772433    2580 scope.go:117] "RemoveContainer" containerID="99a3947d7e94c3948a29e3f4c16b2d71bdf9f69d662e2d49a67e39a664f3a9de"
Jul 20 11:24:37.772727 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:24:37.772708    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a3947d7e94c3948a29e3f4c16b2d71bdf9f69d662e2d49a67e39a664f3a9de\": container with ID starting with 99a3947d7e94c3948a29e3f4c16b2d71bdf9f69d662e2d49a67e39a664f3a9de not found: ID does not exist" containerID="99a3947d7e94c3948a29e3f4c16b2d71bdf9f69d662e2d49a67e39a664f3a9de"
Jul 20 11:24:37.772806 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.772734    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a3947d7e94c3948a29e3f4c16b2d71bdf9f69d662e2d49a67e39a664f3a9de"} err="failed to get container status \"99a3947d7e94c3948a29e3f4c16b2d71bdf9f69d662e2d49a67e39a664f3a9de\": rpc error: code = NotFound desc = could not find container \"99a3947d7e94c3948a29e3f4c16b2d71bdf9f69d662e2d49a67e39a664f3a9de\": container with ID starting with 99a3947d7e94c3948a29e3f4c16b2d71bdf9f69d662e2d49a67e39a664f3a9de not found: ID does not exist"
Jul 20 11:24:37.781638 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.781610    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 20 11:24:37.786863 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:37.786841    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 20 11:24:38.410785 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:38.410750    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbf60339-7cf1-4dd5-8d81-6dcea1605b26" path="/var/lib/kubelet/pods/dbf60339-7cf1-4dd5-8d81-6dcea1605b26/volumes"
Jul 20 11:24:40.786999 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:40.786964    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:24:45.637503 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:45.637471    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:24:45.737060 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:45.737027    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:24:47.403888 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:47.403853    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:24:52.014680 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:52.014594    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:24:53.230800 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:53.230766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:24:53.307670 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:53.307636    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:24:54.803551 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:54.803516    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:24:55.100139 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.100042    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"]
Jul 20 11:24:55.100874 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.100599    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1474ce33-ed5a-4503-a3f9-7e02be141e3a" containerName="storage-initializer"
Jul 20 11:24:55.100874 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.100625    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1474ce33-ed5a-4503-a3f9-7e02be141e3a" containerName="storage-initializer"
Jul 20 11:24:55.100874 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.100655    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf60339-7cf1-4dd5-8d81-6dcea1605b26" containerName="main"
Jul 20 11:24:55.100874 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.100664    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf60339-7cf1-4dd5-8d81-6dcea1605b26" containerName="main"
Jul 20 11:24:55.100874 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.100681    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1474ce33-ed5a-4503-a3f9-7e02be141e3a" containerName="main"
Jul 20 11:24:55.100874 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.100690    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1474ce33-ed5a-4503-a3f9-7e02be141e3a" containerName="main"
Jul 20 11:24:55.100874 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.100777    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1474ce33-ed5a-4503-a3f9-7e02be141e3a" containerName="main"
Jul 20 11:24:55.100874 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.100790    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbf60339-7cf1-4dd5-8d81-6dcea1605b26" containerName="main"
Jul 20 11:24:55.109086 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.109061    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"
Jul 20 11:24:55.111752 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.111725    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 20 11:24:55.111884 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.111726    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-8pr62\""
Jul 20 11:24:55.111884 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.111768    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 20 11:24:55.111884 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.111806    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-s7p25\""
Jul 20 11:24:55.111884 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.111726    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 20 11:24:55.117833 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.117806    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"]
Jul 20 11:24:55.160980 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.160922    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:24:55.242303 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.242266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc\" (UID: \"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"
Jul 20 11:24:55.242492 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.242322    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f8fr\" (UniqueName: \"kubernetes.io/projected/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d-kube-api-access-5f8fr\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc\" (UID: \"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"
Jul 20 11:24:55.242492 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.242413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc\" (UID: \"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"
Jul 20 11:24:55.343846 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.343792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc\" (UID: \"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"
Jul 20 11:24:55.344044 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.343863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f8fr\" (UniqueName: \"kubernetes.io/projected/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d-kube-api-access-5f8fr\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc\" (UID: \"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"
Jul 20 11:24:55.344044 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.343892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc\" (UID: \"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"
Jul 20 11:24:55.344323 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.344298    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc\" (UID: \"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"
Jul 20 11:24:55.346557 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.346536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc\" (UID: \"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"
Jul 20 11:24:55.351875 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.351824    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f8fr\" (UniqueName: \"kubernetes.io/projected/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d-kube-api-access-5f8fr\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc\" (UID: \"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"
Jul 20 11:24:55.421838 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.421797    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"
Jul 20 11:24:55.558749 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.558724    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"]
Jul 20 11:24:55.560624 ip-10-0-133-160 kubenswrapper[2580]: W0720 11:24:55.560582    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3705ab_27fa_4c03_9a51_4a6cd3720e5d.slice/crio-a115f12d237ce788fe4bcdb2dd5d017ac4cf00a76b865b10e48c3d3c42519dbf WatchSource:0}: Error finding container a115f12d237ce788fe4bcdb2dd5d017ac4cf00a76b865b10e48c3d3c42519dbf: Status 404 returned error can't find the container with id a115f12d237ce788fe4bcdb2dd5d017ac4cf00a76b865b10e48c3d3c42519dbf
Jul 20 11:24:55.855927 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.855893    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc" event={"ID":"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d","Type":"ContainerStarted","Data":"5e4e0c62e8f442b4c2f9833e5072866ca0d55db4b9b99dc63667a005160d0fb7"}
Jul 20 11:24:55.856392 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.855948    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc" event={"ID":"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d","Type":"ContainerStarted","Data":"a115f12d237ce788fe4bcdb2dd5d017ac4cf00a76b865b10e48c3d3c42519dbf"}
Jul 20 11:24:55.856392 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.855991    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"
Jul 20 11:24:55.870633 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:55.870510    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc" podStartSLOduration=0.870490421 podStartE2EDuration="870.490421ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:24:55.870347402 +0000 UTC m=+5307.919211049" watchObservedRunningTime="2026-07-20 11:24:55.870490421 +0000 UTC m=+5307.919354074"
Jul 20 11:24:58.173225 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:58.173186    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:24:58.273663 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:24:58.273630    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:25:05.000393 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:05.000327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:25:16.073703 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:16.073667    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:25:16.174072 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:16.174040    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:25:26.865155 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:26.865123    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"
Jul 20 11:25:28.048557 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:28.048518    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"]
Jul 20 11:25:28.049039 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:28.048967    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:25:28.105604 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:28.105571    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:25:29.055938 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:29.055897    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:25:36.865661 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:36.865622    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc" podUID="2a3705ab-27fa-4c03-9a51-4a6cd3720e5d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 11:25:39.851694 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:39.851661    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc" message="" err="command '/bin/sleep 15' exited with 137: "
Jul 20 11:25:39.851694 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:25:39.851693    2580 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc" podUID="2a3705ab-27fa-4c03-9a51-4a6cd3720e5d" containerName="main" containerID="cri-o://5e4e0c62e8f442b4c2f9833e5072866ca0d55db4b9b99dc63667a005160d0fb7"
Jul 20 11:25:39.852109 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:39.851748    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc" podUID="2a3705ab-27fa-4c03-9a51-4a6cd3720e5d" containerName="main" containerID="cri-o://5e4e0c62e8f442b4c2f9833e5072866ca0d55db4b9b99dc63667a005160d0fb7" gracePeriod=49
Jul 20 11:25:39.997511 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:39.997487    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc_2a3705ab-27fa-4c03-9a51-4a6cd3720e5d/main/0.log"
Jul 20 11:25:39.999779 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:39.999760    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"
Jul 20 11:25:40.090253 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.090226    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc_2a3705ab-27fa-4c03-9a51-4a6cd3720e5d/main/0.log"
Jul 20 11:25:40.090452 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.090266    2580 generic.go:356] "Generic (PLEG): container finished" podID="2a3705ab-27fa-4c03-9a51-4a6cd3720e5d" containerID="5e4e0c62e8f442b4c2f9833e5072866ca0d55db4b9b99dc63667a005160d0fb7" exitCode=1
Jul 20 11:25:40.090452 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.090313    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"
Jul 20 11:25:40.090452 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.090352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc" event={"ID":"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d","Type":"ContainerDied","Data":"5e4e0c62e8f442b4c2f9833e5072866ca0d55db4b9b99dc63667a005160d0fb7"}
Jul 20 11:25:40.090452 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.090410    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc" event={"ID":"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d","Type":"ContainerDied","Data":"a115f12d237ce788fe4bcdb2dd5d017ac4cf00a76b865b10e48c3d3c42519dbf"}
Jul 20 11:25:40.090452 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.090437    2580 scope.go:117] "RemoveContainer" containerID="5e4e0c62e8f442b4c2f9833e5072866ca0d55db4b9b99dc63667a005160d0fb7"
Jul 20 11:25:40.099226 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.099210    2580 scope.go:117] "RemoveContainer" containerID="5e4e0c62e8f442b4c2f9833e5072866ca0d55db4b9b99dc63667a005160d0fb7"
Jul 20 11:25:40.099529 ip-10-0-133-160 kubenswrapper[2580]: E0720 11:25:40.099510    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e4e0c62e8f442b4c2f9833e5072866ca0d55db4b9b99dc63667a005160d0fb7\": container with ID starting with 5e4e0c62e8f442b4c2f9833e5072866ca0d55db4b9b99dc63667a005160d0fb7 not found: ID does not exist" containerID="5e4e0c62e8f442b4c2f9833e5072866ca0d55db4b9b99dc63667a005160d0fb7"
Jul 20 11:25:40.099590 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.099543    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e4e0c62e8f442b4c2f9833e5072866ca0d55db4b9b99dc63667a005160d0fb7"} err="failed to get container status \"5e4e0c62e8f442b4c2f9833e5072866ca0d55db4b9b99dc63667a005160d0fb7\": rpc error: code = NotFound desc = could not find container \"5e4e0c62e8f442b4c2f9833e5072866ca0d55db4b9b99dc63667a005160d0fb7\": container with ID starting with 5e4e0c62e8f442b4c2f9833e5072866ca0d55db4b9b99dc63667a005160d0fb7 not found: ID does not exist"
Jul 20 11:25:40.144892 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.144852    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d-tokenizer-uds\") pod \"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d\" (UID: \"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d\") "
Jul 20 11:25:40.145099 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.144910    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d-tls-certs\") pod \"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d\" (UID: \"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d\") "
Jul 20 11:25:40.145099 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.144947    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f8fr\" (UniqueName: \"kubernetes.io/projected/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d-kube-api-access-5f8fr\") pod \"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d\" (UID: \"2a3705ab-27fa-4c03-9a51-4a6cd3720e5d\") "
Jul 20 11:25:40.145261 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.145229    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2a3705ab-27fa-4c03-9a51-4a6cd3720e5d" (UID: "2a3705ab-27fa-4c03-9a51-4a6cd3720e5d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:25:40.147337 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.147300    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d-kube-api-access-5f8fr" (OuterVolumeSpecName: "kube-api-access-5f8fr") pod "2a3705ab-27fa-4c03-9a51-4a6cd3720e5d" (UID: "2a3705ab-27fa-4c03-9a51-4a6cd3720e5d"). InnerVolumeSpecName "kube-api-access-5f8fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:25:40.147337 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.147307    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a3705ab-27fa-4c03-9a51-4a6cd3720e5d" (UID: "2a3705ab-27fa-4c03-9a51-4a6cd3720e5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:25:40.246311 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.246269    2580 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d-tokenizer-uds\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:25:40.246311 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.246310    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d-tls-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:25:40.246311 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.246319    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f8fr\" (UniqueName: \"kubernetes.io/projected/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d-kube-api-access-5f8fr\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 20 11:25:40.411266 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.411237    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"]
Jul 20 11:25:40.416792 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:40.416758    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d996894nlc"]
Jul 20 11:25:42.411259 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:42.411225    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a3705ab-27fa-4c03-9a51-4a6cd3720e5d" path="/var/lib/kubelet/pods/2a3705ab-27fa-4c03-9a51-4a6cd3720e5d/volumes"
Jul 20 11:25:48.076059 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:48.076021    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:25:49.531995 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:49.531960    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:25:49.633428 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:49.633392    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:25:53.851560 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:53.851521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:25:59.571180 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:25:59.571130    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:26:01.534287 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:01.534253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:26:12.555385 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:12.555344    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zltzb_6e9300ef-83e4-4b50-947f-06b61d0ec232/discovery/0.log"
Jul 20 11:26:12.567843 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:12.567814    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-jfpwp_06712755-7e3d-4b4c-9c07-3260be21366b/istio-proxy/0.log"
Jul 20 11:26:12.578370 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:12.578343    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-jfpwp_06712755-7e3d-4b4c-9c07-3260be21366b/istio-proxy/1.log"
Jul 20 11:26:13.404653 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:13.404612    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zltzb_6e9300ef-83e4-4b50-947f-06b61d0ec232/discovery/0.log"
Jul 20 11:26:13.422442 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:13.422414    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-jfpwp_06712755-7e3d-4b4c-9c07-3260be21366b/istio-proxy/0.log"
Jul 20 11:26:13.433295 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:13.433268    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-jfpwp_06712755-7e3d-4b4c-9c07-3260be21366b/istio-proxy/1.log"
Jul 20 11:26:14.163597 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:14.163565    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-kxw42_76ff9bc1-31e3-4adf-b705-42b53b9c122c/manager/0.log"
Jul 20 11:26:14.376504 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:14.376469    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-q97v8_5c67341c-9eb9-4b65-9661-a5dd48e6ca74/limitador/0.log"
Jul 20 11:26:14.394280 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:14.394253    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rvhvm_95ce9651-daa6-4848-aedb-dd91397e2d6f/manager/0.log"
Jul 20 11:26:20.304890 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:20.304816    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:26:20.447016 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:20.446988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:26:20.507444 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:20.507411    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:26:23.463913 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:23.463879    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2tscg_8fc245f5-c152-44de-ad20-dd240d3741bb/global-pull-secret-syncer/0.log"
Jul 20 11:26:23.575902 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:23.575861    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log"
Jul 20 11:26:23.711683 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:23.711650    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-160.ec2.internal_093884edf7fa6418bd467973eecef80e/haproxy/0.log"
Jul 20 11:26:26.152927 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:26.152888    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:26:26.224424 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:26.224391    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:26:28.310490 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:28.310457    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-kxw42_76ff9bc1-31e3-4adf-b705-42b53b9c122c/manager/0.log"
Jul 20 11:26:28.551086 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:28.551055    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-q97v8_5c67341c-9eb9-4b65-9661-a5dd48e6ca74/limitador/0.log"
Jul 20 11:26:28.580344 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:28.580254    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rvhvm_95ce9651-daa6-4848-aedb-dd91397e2d6f/manager/0.log"
Jul 20 11:26:30.221051 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:30.221016    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-rd75s_4c1f99dc-12d6-4019-ae66-0d06db01e2a2/cluster-monitoring-operator/0.log"
Jul 20 11:26:30.397404 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:30.397376    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4wwp_9fc14519-d15a-420e-a8af-06a2babcdc4b/node-exporter/0.log"
Jul 20 11:26:30.425894 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:30.425862    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4wwp_9fc14519-d15a-420e-a8af-06a2babcdc4b/kube-rbac-proxy/0.log"
Jul 20 11:26:30.457744 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:30.457721    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4wwp_9fc14519-d15a-420e-a8af-06a2babcdc4b/init-textfile/0.log"
Jul 20 11:26:31.067181 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:31.067143    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-786b8f867b-bmpv5_bb35413a-c78d-4e16-8355-ae387bcb9260/telemeter-client/0.log"
Jul 20 11:26:31.093253 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:31.093227    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-786b8f867b-bmpv5_bb35413a-c78d-4e16-8355-ae387bcb9260/reload/0.log"
Jul 20 11:26:31.120875 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:31.120849    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-786b8f867b-bmpv5_bb35413a-c78d-4e16-8355-ae387bcb9260/kube-rbac-proxy/0.log"
Jul 20 11:26:34.111073 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.111011    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"]
Jul 20 11:26:34.111662 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.111632    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a3705ab-27fa-4c03-9a51-4a6cd3720e5d" containerName="main"
Jul 20 11:26:34.111662 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.111661    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a3705ab-27fa-4c03-9a51-4a6cd3720e5d" containerName="main"
Jul 20 11:26:34.111875 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.111758    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a3705ab-27fa-4c03-9a51-4a6cd3720e5d" containerName="main"
Jul 20 11:26:34.116211 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.116191    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:34.118044 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.118026    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4rtcj\"/\"default-dockercfg-dh72t\""
Jul 20 11:26:34.118135 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.118028    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4rtcj\"/\"openshift-service-ca.crt\""
Jul 20 11:26:34.118514 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.118499    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4rtcj\"/\"kube-root-ca.crt\""
Jul 20 11:26:34.123418 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.123181    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"]
Jul 20 11:26:34.183296 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.183250    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2fd792f3-e3d9-40ad-a694-b9605901d7c1-podres\") pod \"perf-node-gather-daemonset-rtdpc\" (UID: \"2fd792f3-e3d9-40ad-a694-b9605901d7c1\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:34.183296 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.183302    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fd792f3-e3d9-40ad-a694-b9605901d7c1-lib-modules\") pod \"perf-node-gather-daemonset-rtdpc\" (UID: \"2fd792f3-e3d9-40ad-a694-b9605901d7c1\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:34.183556 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.183338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rdtq\" (UniqueName: \"kubernetes.io/projected/2fd792f3-e3d9-40ad-a694-b9605901d7c1-kube-api-access-5rdtq\") pod \"perf-node-gather-daemonset-rtdpc\" (UID: \"2fd792f3-e3d9-40ad-a694-b9605901d7c1\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:34.183556 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.183385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2fd792f3-e3d9-40ad-a694-b9605901d7c1-proc\") pod \"perf-node-gather-daemonset-rtdpc\" (UID: \"2fd792f3-e3d9-40ad-a694-b9605901d7c1\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:34.183556 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.183477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fd792f3-e3d9-40ad-a694-b9605901d7c1-sys\") pod \"perf-node-gather-daemonset-rtdpc\" (UID: \"2fd792f3-e3d9-40ad-a694-b9605901d7c1\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:34.284083 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.284029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2fd792f3-e3d9-40ad-a694-b9605901d7c1-proc\") pod \"perf-node-gather-daemonset-rtdpc\" (UID: \"2fd792f3-e3d9-40ad-a694-b9605901d7c1\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:34.284302 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.284119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fd792f3-e3d9-40ad-a694-b9605901d7c1-sys\") pod \"perf-node-gather-daemonset-rtdpc\" (UID: \"2fd792f3-e3d9-40ad-a694-b9605901d7c1\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:34.284302 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.284142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2fd792f3-e3d9-40ad-a694-b9605901d7c1-podres\") pod \"perf-node-gather-daemonset-rtdpc\" (UID: \"2fd792f3-e3d9-40ad-a694-b9605901d7c1\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:34.284302 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.284186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fd792f3-e3d9-40ad-a694-b9605901d7c1-lib-modules\") pod \"perf-node-gather-daemonset-rtdpc\" (UID: \"2fd792f3-e3d9-40ad-a694-b9605901d7c1\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:34.284302 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.284161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2fd792f3-e3d9-40ad-a694-b9605901d7c1-proc\") pod \"perf-node-gather-daemonset-rtdpc\" (UID: \"2fd792f3-e3d9-40ad-a694-b9605901d7c1\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:34.284302 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.284208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rdtq\" (UniqueName: \"kubernetes.io/projected/2fd792f3-e3d9-40ad-a694-b9605901d7c1-kube-api-access-5rdtq\") pod \"perf-node-gather-daemonset-rtdpc\" (UID: \"2fd792f3-e3d9-40ad-a694-b9605901d7c1\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:34.284302 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.284251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fd792f3-e3d9-40ad-a694-b9605901d7c1-sys\") pod \"perf-node-gather-daemonset-rtdpc\" (UID: \"2fd792f3-e3d9-40ad-a694-b9605901d7c1\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:34.284525 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.284307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fd792f3-e3d9-40ad-a694-b9605901d7c1-lib-modules\") pod \"perf-node-gather-daemonset-rtdpc\" (UID: \"2fd792f3-e3d9-40ad-a694-b9605901d7c1\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:34.284525 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.284311    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2fd792f3-e3d9-40ad-a694-b9605901d7c1-podres\") pod \"perf-node-gather-daemonset-rtdpc\" (UID: \"2fd792f3-e3d9-40ad-a694-b9605901d7c1\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:34.292537 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.292503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rdtq\" (UniqueName: \"kubernetes.io/projected/2fd792f3-e3d9-40ad-a694-b9605901d7c1-kube-api-access-5rdtq\") pod \"perf-node-gather-daemonset-rtdpc\" (UID: \"2fd792f3-e3d9-40ad-a694-b9605901d7c1\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:34.427099 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.427012    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:34.566723 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:34.566697    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"]
Jul 20 11:26:34.569114 ip-10-0-133-160 kubenswrapper[2580]: W0720 11:26:34.569080    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2fd792f3_e3d9_40ad_a694_b9605901d7c1.slice/crio-9e57ee930e8ccc84ff92e1d26d39e64b5a614d87c8991a04acc5466c472c5c99 WatchSource:0}: Error finding container 9e57ee930e8ccc84ff92e1d26d39e64b5a614d87c8991a04acc5466c472c5c99: Status 404 returned error can't find the container with id 9e57ee930e8ccc84ff92e1d26d39e64b5a614d87c8991a04acc5466c472c5c99
Jul 20 11:26:35.364342 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:35.364304    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc" event={"ID":"2fd792f3-e3d9-40ad-a694-b9605901d7c1","Type":"ContainerStarted","Data":"3de35d919fdadeae0b2e268c27bf4e187a3cbe98fe8326fa3e4b00e0fc18c6af"}
Jul 20 11:26:35.364804 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:35.364362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc" event={"ID":"2fd792f3-e3d9-40ad-a694-b9605901d7c1","Type":"ContainerStarted","Data":"9e57ee930e8ccc84ff92e1d26d39e64b5a614d87c8991a04acc5466c472c5c99"}
Jul 20 11:26:35.364804 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:35.364391    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:35.381849 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:35.381794    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc" podStartSLOduration=1.381777881 podStartE2EDuration="1.381777881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 11:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:26:35.379530161 +0000 UTC m=+5407.428393810" watchObservedRunningTime="2026-07-20 11:26:35.381777881 +0000 UTC m=+5407.430641579"
Jul 20 11:26:35.476508 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:35.476478    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2hnhp_8215ffa1-8dfa-470e-be51-643f5adbc99c/dns/0.log"
Jul 20 11:26:35.499600 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:35.499575    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2hnhp_8215ffa1-8dfa-470e-be51-643f5adbc99c/kube-rbac-proxy/0.log"
Jul 20 11:26:35.671074 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:35.670995    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jfd48_bf6c7a6e-c225-431b-9348-d50084b9d1fe/dns-node-resolver/0.log"
Jul 20 11:26:36.274872 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:36.274841    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g2gl7_fc481244-2a08-4f08-a199-7d02c4eaa9fc/node-ca/0.log"
Jul 20 11:26:37.339946 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:37.339905    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zltzb_6e9300ef-83e4-4b50-947f-06b61d0ec232/discovery/0.log"
Jul 20 11:26:37.367060 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:37.367020    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-jfpwp_06712755-7e3d-4b4c-9c07-3260be21366b/istio-proxy/0.log"
Jul 20 11:26:37.378561 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:37.378531    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-jfpwp_06712755-7e3d-4b4c-9c07-3260be21366b/istio-proxy/1.log"
Jul 20 11:26:37.503025 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:37.502994    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42cdx_35cc8bd5-0ca5-4fcf-ab59-61ac94b4d420/konnectivity-agent/0.log" limit=1024
Jul 20 11:26:37.939412 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:37.939382    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nn2gp_a84537ba-7e16-4295-8b3c-818c578c58fa/serve-healthcheck-canary/0.log"
Jul 20 11:26:38.481635 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:38.481604    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-rm45p_944513c3-bdd1-40f5-8ffd-bd72d60e2d50/insights-operator/0.log"
Jul 20 11:26:38.482918 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:38.482897    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-rm45p_944513c3-bdd1-40f5-8ffd-bd72d60e2d50/insights-operator/1.log"
Jul 20 11:26:38.571860 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:38.571811    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h8tvt_540eee1e-9d83-40b2-afb6-970f4df13724/kube-rbac-proxy/0.log"
Jul 20 11:26:38.592949 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:38.592921    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h8tvt_540eee1e-9d83-40b2-afb6-970f4df13724/exporter/0.log"
Jul 20 11:26:38.614800 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:38.614778    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h8tvt_540eee1e-9d83-40b2-afb6-970f4df13724/extractor/0.log"
Jul 20 11:26:41.380755 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:41.380729    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-rtdpc"
Jul 20 11:26:43.288602 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:43.288565    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-bn4s5_d774081a-91db-4bd2-9f8c-5616fc11d52a/seaweedfs/0.log"
Jul 20 11:26:43.714961 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:43.714926    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zltzb_6e9300ef-83e4-4b50-947f-06b61d0ec232/discovery/0.log"
Jul 20 11:26:43.730880 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:43.730843    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-jfpwp_06712755-7e3d-4b4c-9c07-3260be21366b/istio-proxy/0.log"
Jul 20 11:26:43.741907 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:43.741833    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-jfpwp_06712755-7e3d-4b4c-9c07-3260be21366b/istio-proxy/1.log"
Jul 20 11:26:44.964684 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:44.964647    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zltzb_6e9300ef-83e4-4b50-947f-06b61d0ec232/discovery/0.log"
Jul 20 11:26:44.980304 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:44.980278    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-jfpwp_06712755-7e3d-4b4c-9c07-3260be21366b/istio-proxy/0.log"
Jul 20 11:26:44.990952 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:44.990929    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-jfpwp_06712755-7e3d-4b4c-9c07-3260be21366b/istio-proxy/1.log"
Jul 20 11:26:48.775094 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:48.775056    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-5npqn_1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6/kube-storage-version-migrator-operator/1.log"
Jul 20 11:26:48.775967 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:48.775950    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-5npqn_1f34d656-cf4c-4b3a-aa22-ba342ba8cbe6/kube-storage-version-migrator-operator/0.log"
Jul 20 11:26:50.077314 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:50.077288    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ww8gr_17f16e10-bfe7-4dbf-89c3-9b4d68077838/kube-multus-additional-cni-plugins/0.log"
Jul 20 11:26:50.099354 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:50.099319    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ww8gr_17f16e10-bfe7-4dbf-89c3-9b4d68077838/egress-router-binary-copy/0.log"
Jul 20 11:26:50.120949 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:50.120916    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ww8gr_17f16e10-bfe7-4dbf-89c3-9b4d68077838/cni-plugins/0.log"
Jul 20 11:26:50.145026 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:50.145000    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ww8gr_17f16e10-bfe7-4dbf-89c3-9b4d68077838/bond-cni-plugin/0.log"
Jul 20 11:26:50.167876 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:50.167851    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ww8gr_17f16e10-bfe7-4dbf-89c3-9b4d68077838/routeoverride-cni/0.log"
Jul 20 11:26:50.189836 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:50.189805    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ww8gr_17f16e10-bfe7-4dbf-89c3-9b4d68077838/whereabouts-cni-bincopy/0.log"
Jul 20 11:26:50.210886 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:50.210855    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ww8gr_17f16e10-bfe7-4dbf-89c3-9b4d68077838/whereabouts-cni/0.log"
Jul 20 11:26:50.303346 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:50.303311    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qnvtm_224aadf9-e413-4791-8f19-1950846b8374/kube-multus/0.log"
Jul 20 11:26:50.403231 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:50.403133    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jms2f_5e4c9043-3e1d-49f1-8550-ce507005bcc4/network-metrics-daemon/0.log"
Jul 20 11:26:50.424016 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:50.423983    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jms2f_5e4c9043-3e1d-49f1-8550-ce507005bcc4/kube-rbac-proxy/0.log"
Jul 20 11:26:51.876929 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:51.876888    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg8lz_8245a5b2-d969-4790-adb3-c9d22661cc81/ovn-controller/0.log"
Jul 20 11:26:51.893716 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:51.893690    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg8lz_8245a5b2-d969-4790-adb3-c9d22661cc81/ovn-acl-logging/0.log"
Jul 20 11:26:51.920972 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:51.920943    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg8lz_8245a5b2-d969-4790-adb3-c9d22661cc81/kube-rbac-proxy-node/0.log"
Jul 20 11:26:51.943655 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:51.943629    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg8lz_8245a5b2-d969-4790-adb3-c9d22661cc81/kube-rbac-proxy-ovn-metrics/0.log"
Jul 20 11:26:51.964033 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:51.964005    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg8lz_8245a5b2-d969-4790-adb3-c9d22661cc81/northd/0.log"
Jul 20 11:26:51.985309 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:51.985288    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg8lz_8245a5b2-d969-4790-adb3-c9d22661cc81/nbdb/0.log"
Jul 20 11:26:52.008233 ip-10-0-133-160 kubenswrapper[2580]: I0720 11:26:52.008208    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg8lz_8245a5b2-d969-4790-adb3-c9d22661cc81/sbdb/0.log"