Jul 29 22:46:16.218277 ip-10-0-128-242 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 22:46:16.612749 ip-10-0-128-242 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 22:46:16.612749 ip-10-0-128-242 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 22:46:16.612749 ip-10-0-128-242 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 22:46:16.612749 ip-10-0-128-242 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 22:46:16.612749 ip-10-0-128-242 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 22:46:16.613750 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.613652    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 22:46:16.618567 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618552    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 22:46:16.618567 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618566    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618571    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618574    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618578    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618581    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618584    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618587    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618592    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618595    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618603    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618606    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618608    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618611    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618614    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618617    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618619    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618622    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618625    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618627    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618630    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 22:46:16.618631 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618633    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 22:46:16.619164 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618637    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 22:46:16.619164 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618641    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 22:46:16.619164 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618645    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 22:46:16.619164 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618648    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 22:46:16.619164 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618651    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 22:46:16.619164 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618654    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 22:46:16.619164 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618657    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 22:46:16.619164 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618660    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 22:46:16.619164 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618664    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 22:46:16.619164 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618668    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 22:46:16.619164 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618671    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 22:46:16.619164 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618674    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 22:46:16.619164 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618676    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 22:46:16.619164 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618680    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 22:46:16.619164 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618682    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 22:46:16.619164 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618686    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 22:46:16.619164 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618689    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 22:46:16.619164 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618691    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618695    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618698    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618700    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618703    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618705    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618708    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618710    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618713    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618715    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618718    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618720    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618723    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618725    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618728    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618730    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618733    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618736    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618739    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618741    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 22:46:16.619588 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618744    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618746    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618749    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618751    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618754    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618756    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618759    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618761    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618764    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618767    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618769    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618772    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618774    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618777    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618780    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618783    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618785    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618788    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618790    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618793    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 22:46:16.620096 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618795    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618798    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618800    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618803    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618805    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618808    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618811    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618813    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618816    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618819    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618821    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618824    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618827    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618829    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618832    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618834    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618836    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618839    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618842    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.618845    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 22:46:16.620572 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619541    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619549    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619552    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619555    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619558    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619561    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619563    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619566    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619569    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619571    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619574    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619576    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619579    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619582    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619585    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619588    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619591    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619593    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619596    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 22:46:16.621066 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619600    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619606    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619609    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619613    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619616    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619619    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619621    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619624    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619627    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619629    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619632    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619635    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619637    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619642    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619645    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619648    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619651    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619654    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619656    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619659    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 22:46:16.621564 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619662    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619664    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619667    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619670    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619672    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619675    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619677    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619680    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619682    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619685    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619688    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619690    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619693    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619697    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619700    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619703    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619705    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619708    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619710    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 22:46:16.622059 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619713    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619717    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619720    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619723    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619726    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619728    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619731    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619738    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619742    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619744    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619747    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619750    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619752    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619756    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619758    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619761    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619764    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619766    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619769    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 22:46:16.622526 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619771    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619774    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619777    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619779    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619782    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619784    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619787    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619790    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619793    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619795    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619798    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619801    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619803    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619806    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619809    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619811    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619814    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619816    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619819    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619821    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 22:46:16.623008 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619824    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.619827    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620186    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620191    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620194    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620197    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620199    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620202    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620205    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620208    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620210    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620213    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620216    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620219    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620223    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620227    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620230    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620234    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620237    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 22:46:16.623497 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620240    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620243    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620246    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620249    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620252    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620256    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620258    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620261    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620263    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620266    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620268    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620271    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620273    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620276    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620280    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620283    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620285    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620288    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620291    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 22:46:16.623948 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620293    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620296    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620298    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620301    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620304    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620307    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620309    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620312    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620315    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620317    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620320    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620322    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620325    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620328    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620330    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620333    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620336    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620338    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620341    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620343    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 22:46:16.624422 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620346    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620349    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620351    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620353    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620356    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620358    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620361    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620366    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620370    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620372    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620375    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620377    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620380    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620383    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620385    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620387    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620390    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620394    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620397    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620399    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 22:46:16.624893 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620402    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620404    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620407    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620409    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620412    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620416    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620419    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620421    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620424    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620426    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620429    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620431    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620434    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620438    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620442    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620445    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620448    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620450    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620453    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620457    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 22:46:16.625386 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620459    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620462    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.620465    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620531    2581 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620538    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620544    2581 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620548    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620552    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620555    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620559    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620563    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620566    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620569    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620573    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620576    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620579    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620582    2581 flags.go:64] FLAG: --cgroup-root=""
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620587    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620590    2581 flags.go:64] FLAG: --client-ca-file=""
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620593    2581 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620596    2581 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620601    2581 flags.go:64] FLAG: --cluster-domain=""
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620603    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620607    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 22:46:16.625886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620610    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620613    2581 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620617    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620620    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620623    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620626    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620630    2581 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620633    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620636    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620639    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620643    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620647    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620650    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620653    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620656    2581 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620659    2581 flags.go:64] FLAG: --enable-server="true"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620662    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620666    2581 flags.go:64] FLAG: --event-burst="100"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620669    2581 flags.go:64] FLAG: --event-qps="50"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620672    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620676    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620679    2581 flags.go:64] FLAG: --eviction-hard=""
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620682    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620685    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620688    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 22:46:16.626518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620692    2581 flags.go:64] FLAG: --eviction-soft=""
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620695    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620698    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620701    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620704    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620707    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620710    2581 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620713    2581 flags.go:64] FLAG: --feature-gates=""
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620716    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620719    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620722    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620725    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620728    2581 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620731    2581 flags.go:64] FLAG: --help="false"
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620734    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-128-242.ec2.internal"
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620738    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620741    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620744    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620749    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620752    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620755    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620758    2581 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620761    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 22:46:16.627184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620764    2581 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620767    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620770    2581 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620773    2581 flags.go:64] FLAG: --kube-reserved=""
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620776    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620779    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620782    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620785    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620788    2581 flags.go:64] FLAG: --lock-file=""
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620793    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620796    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620799    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620804    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620807    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620810    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620813    2581 flags.go:64] FLAG: --logging-format="text"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620816    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620819    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620822    2581 flags.go:64] FLAG: --manifest-url=""
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620825    2581 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620829    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620832    2581 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620836    2581 flags.go:64] FLAG: --max-pods="110"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620840    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620843    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 22:46:16.628119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620846    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620849    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620852    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620856    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620859    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620865    2581 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620869    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620872    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620875    2581 flags.go:64] FLAG: --pod-cidr=""
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620878    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620888    2581 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620891    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620894    2581 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620897    2581 flags.go:64] FLAG: --port="10250"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620900    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620903    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a8841f33ec60e420"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620908    2581 flags.go:64] FLAG: --qos-reserved=""
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620911    2581 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620914    2581 flags.go:64] FLAG: --register-node="true"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620917    2581 flags.go:64] FLAG: --register-with-taints=""
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620920    2581 flags.go:64] FLAG: --registry-burst="10"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620923    2581 flags.go:64] FLAG: --registry-qps="5"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620926    2581 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620929    2581 flags.go:64] FLAG: --reserved-memory=""
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620932    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 22:46:16.628796 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620935    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620939    2581 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620941    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620944    2581 flags.go:64] FLAG: --runonce="false"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620947    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620950    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620953    2581 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620956    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620959    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620962    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620965    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620968    2581 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620971    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620974    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620977    2581 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620980    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620983    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620986    2581 flags.go:64] FLAG: --system-cgroups=""
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620989    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620994    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.620997    2581 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.621000    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.621004    2581 flags.go:64] FLAG: --tls-min-version=""
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.621008    2581 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.621010    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 22:46:16.629411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.621013    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.621016    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.621019    2581 flags.go:64] FLAG: --v="2"
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.621038    2581 flags.go:64] FLAG: --version="false"
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.621044    2581 flags.go:64] FLAG: --vmodule=""
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.621050    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.621054    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621160    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621163    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621166    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621169    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621172    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621176    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621181    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621184    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621187    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621192    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621196    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621199    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621202    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621205    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 22:46:16.630012 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621207    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621210    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621214    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621218    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621221    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621224    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621227    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621229    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621232    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621236    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621239    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621241    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621244    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621248    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621251    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621254    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621256    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621259    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621262    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 22:46:16.630619 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621264    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621267    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621269    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621272    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621275    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621279    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621282    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621284    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621287    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621290    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621293    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621295    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621298    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621301    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621304    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621306    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621309    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621311    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621314    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621317    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 22:46:16.631130 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621320    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621322    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621326    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621329    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621331    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621334    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621336    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621339    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621341    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621344    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621347    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621350    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621352    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621354    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621357    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621360    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621362    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621366    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621369    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621371    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 22:46:16.631610 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621374    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621377    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621379    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621382    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621385    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621387    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621390    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621392    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621395    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621398    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621400    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621403    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621405    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621408    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621411    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621414    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621416    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621419    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621421    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621424    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 22:46:16.632107 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621427    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 22:46:16.632583 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621429    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 22:46:16.632583 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621432    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 22:46:16.632583 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621435    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 22:46:16.632583 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621437    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 22:46:16.632583 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.621440    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 22:46:16.632583 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.621445    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 22:46:16.632583 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.628995    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 22:46:16.632583 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.629009    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 22:46:16.632583 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.630568    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 22:46:16.632583 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.630578    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 22:46:16.632583 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630635    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 22:46:16.632583 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630640    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 22:46:16.632583 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630643    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 22:46:16.632583 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630646    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 22:46:16.632583 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630649    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630652    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630654    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630657    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630660    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630662    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630665    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630668    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630670    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630673    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630676    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630679    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630681    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630683    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630686    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630688    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630691    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630693    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630696    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630699    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 22:46:16.632946 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630702    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630705    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630709    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630712    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630716    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630718    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630721    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630723    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630726    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630729    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630731    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630734    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630736    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630739    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630741    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630744    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630747    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630750    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630752    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630755    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 22:46:16.633438 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630757    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630760    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630762    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630765    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630768    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630770    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630773    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630775    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630778    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630781    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630783    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630786    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630788    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630793    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630797    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630800    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630802    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630805    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630808    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630810    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 22:46:16.633911 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630813    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630815    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630818    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630820    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630822    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630825    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630827    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630830    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630833    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630835    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630837    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630840    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630842    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630845    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630847    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630851    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630854    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630858    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630861    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 22:46:16.634444 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630864    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 22:46:16.634898 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630867    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 22:46:16.634898 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630870    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 22:46:16.634898 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630873    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 22:46:16.634898 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630876    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 22:46:16.634898 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630879    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 22:46:16.634898 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630881    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 22:46:16.634898 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630884    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 22:46:16.634898 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630886    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 22:46:16.634898 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630889    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 22:46:16.634898 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630892    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 22:46:16.634898 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630894    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 22:46:16.634898 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630897    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 22:46:16.634898 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630899    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 22:46:16.634898 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630902    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 22:46:16.634898 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.630905    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 22:46:16.635317 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.630910    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 22:46:16.635317 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631055    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 22:46:16.635317 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631061    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 22:46:16.635317 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631066    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 22:46:16.635317 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631068    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 22:46:16.635317 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631071    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 22:46:16.635317 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631073    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 22:46:16.635317 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631076    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 22:46:16.635317 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631078    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 22:46:16.635317 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631081    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 22:46:16.635317 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631084    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 22:46:16.635317 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631086    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 22:46:16.635317 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631089    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 22:46:16.635317 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631092    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631096    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631100    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631103    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631106    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631109    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631112    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631115    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631118    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631120    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631124    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631126    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631129    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631132    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631135    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631138    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631140    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631143    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631146    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 22:46:16.635660 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631150    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631153    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631156    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631159    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631161    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631164    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631166    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631169    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631172    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631174    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631177    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631179    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631182    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631185    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631188    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631191    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631193    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631196    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631198    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631201    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 22:46:16.636175 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631203    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631206    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631208    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631211    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631214    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631216    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631219    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631221    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631224    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631226    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631229    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631231    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631234    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631236    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631239    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631241    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631243    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631246    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631248    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631251    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 22:46:16.636638 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631253    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631256    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631258    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631263    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631268    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631271    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631274    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631277    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631280    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631283    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631286    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631290    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631293    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631296    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631298    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631301    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631304    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631307    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631309    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 22:46:16.637112 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631312    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 22:46:16.637587 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631315    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 22:46:16.637587 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631318    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 22:46:16.637587 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631320    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 22:46:16.637587 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631323    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 22:46:16.637587 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631325    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 22:46:16.637587 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631328    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 22:46:16.637587 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631331    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 22:46:16.637587 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:16.631333    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 22:46:16.637587 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.631338    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 22:46:16.637587 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.631459    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 22:46:16.637587 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.635881    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 22:46:16.637587 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.637245    2581 server.go:1019] "Starting client certificate rotation"
Jul 29 22:46:16.637587 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.637353    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 22:46:16.637587 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.637386    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 22:46:16.639559 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.639546    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 22:46:16.640713 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.640696    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 22:46:16.649832 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.649816    2581 log.go:25] "Validated CRI v1 runtime API"
Jul 29 22:46:16.651852 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.651837    2581 log.go:25] "Validated CRI v1 image API"
Jul 29 22:46:16.652561 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.652546    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 22:46:16.654944 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.654926    2581 fs.go:135] Filesystem UUIDs: map[47809ddd-1b00-427d-9e6c-3c8cb19016cb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cc5a13b7-8a8b-468d-affe-1fae019fcc05:/dev/nvme0n1p4]
Jul 29 22:46:16.655042 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.654943    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 22:46:16.660231 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.660121    2581 manager.go:216] Machine: {Timestamp:2026-07-29 22:46:16.658965603 +0000 UTC m=+0.329082823 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103529 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28e0b0326c10b431e172c072c8405f SystemUUID:ec28e0b0-326c-10b4-31e1-72c072c8405f BootID:aef003e2-978a-4181-8294-8919eaac8e51 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3d:e2:32:ee:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3d:e2:32:ee:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:10:9b:3a:90:c3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 22:46:16.660231 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.660224    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 22:46:16.660342 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.660287    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 22:46:16.661697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.661635    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 22:46:16.662539 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.661700    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-242.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 22:46:16.663566 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.663551    2581 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 22:46:16.663639 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.663571    2581 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 22:46:16.663639 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.663586    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 22:46:16.664416 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.664403    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 22:46:16.665297 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.665283    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 22:46:16.667931 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.667917    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 22:46:16.668073 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.668062    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 22:46:16.668132 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.668102    2581 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 22:46:16.668132 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.668116    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 22:46:16.668228 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.668135    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 22:46:16.668228 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.668147    2581 kubelet.go:398] "Adding apiserver pod source"
Jul 29 22:46:16.668228 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.668159    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 22:46:16.668228 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.668158    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 22:46:16.668524 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.668511    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 22:46:16.668584 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.668534    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 22:46:16.670975 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.670959    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 22:46:16.672907 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.672893    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 22:46:16.672962 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.672924    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 22:46:16.675089 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.675077    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 22:46:16.675149 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.675094    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 22:46:16.675149 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.675100    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 22:46:16.675149 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.675106    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 22:46:16.675149 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.675114    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 22:46:16.675149 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.675119    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 22:46:16.675149 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.675127    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 22:46:16.675149 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.675132    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 22:46:16.675149 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.675138    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 22:46:16.675149 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.675144    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 22:46:16.675149 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.675152    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 22:46:16.675395 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.675158    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 22:46:16.675903 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.675892    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 22:46:16.675940 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.675905    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 22:46:16.677161 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:16.677139    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 22:46:16.677161 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:16.677148    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 22:46:16.679061 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.679048    2581 server.go:1268] "Started kubelet"
Jul 29 22:46:16.679121 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.679094    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 22:46:16.679217 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.679179    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 22:46:16.679278 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.679267    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 22:46:16.679777 ip-10-0-128-242 systemd[1]: Started Kubernetes Kubelet.
Jul 29 22:46:16.679948 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.679929    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-242.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 22:46:16.681607 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.681582    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 22:46:16.682006 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.681988    2581 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 22:46:16.687423 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:16.687407    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 22:46:16.688128 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:16.686602    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-242.ec2.internal.18c6e42b3e4dd67f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-242.ec2.internal,UID:ip-10-0-128-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-242.ec2.internal,},FirstTimestamp:2026-07-29 22:46:16.678995583 +0000 UTC m=+0.349112806,LastTimestamp:2026-07-29 22:46:16.678995583 +0000 UTC m=+0.349112806,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-242.ec2.internal,}"
Jul 29 22:46:16.688961 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.688944    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 22:46:16.689060 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.688965    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 22:46:16.689060 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:16.689013    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-242.ec2.internal\" not found"
Jul 29 22:46:16.689905 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.689885    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 22:46:16.689996 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.689886    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 22:46:16.689996 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.689987    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 22:46:16.690176 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.690043    2581 factory.go:55] Registering systemd factory
Jul 29 22:46:16.690176 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.690056    2581 factory.go:223] Registration of the systemd container factory successfully
Jul 29 22:46:16.690176 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.690124    2581 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 22:46:16.690176 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.690134    2581 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 22:46:16.690334 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.690298    2581 factory.go:153] Registering CRI-O factory
Jul 29 22:46:16.690334 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.690311    2581 factory.go:223] Registration of the crio container factory successfully
Jul 29 22:46:16.690397 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.690350    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 22:46:16.690397 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.690368    2581 factory.go:103] Registering Raw factory
Jul 29 22:46:16.690397 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.690382    2581 manager.go:1196] Started watching for new ooms in manager
Jul 29 22:46:16.690871 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.690853    2581 manager.go:349] Starting recovery of all containers
Jul 29 22:46:16.692517 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:16.692466    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 22:46:16.692619 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:16.692567    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 22:46:16.702057 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.702037    2581 manager.go:354] Recovery completed
Jul 29 22:46:16.706317 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.706304    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:16.707833 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.707809    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:16.707889 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.707845    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:16.707889 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.707856    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:16.708323 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.708306    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 22:46:16.708323 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.708321    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 22:46:16.708450 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.708337    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 22:46:16.708450 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.708322    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:16.709478 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.709456    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:16.709554 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.709487    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:16.709554 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.709501    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:16.710230 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:16.710130    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-242.ec2.internal.18c6e42b4005da0d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-242.ec2.internal,UID:ip-10-0-128-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-242.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-242.ec2.internal,},FirstTimestamp:2026-07-29 22:46:16.707832333 +0000 UTC m=+0.377949553,LastTimestamp:2026-07-29 22:46:16.707832333 +0000 UTC m=+0.377949553,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-242.ec2.internal,}"
Jul 29 22:46:16.711621 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.711607    2581 policy_none.go:49] "None policy: Start"
Jul 29 22:46:16.711692 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.711625    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 22:46:16.711692 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.711638    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 22:46:16.712757 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.712745    2581 policy_none.go:47] "Start"
Jul 29 22:46:16.719829 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.719811    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rn5ng"
Jul 29 22:46:16.720732 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:16.720675    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-242.ec2.internal.18c6e42b40062134  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-242.ec2.internal,UID:ip-10-0-128-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-242.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-242.ec2.internal,},FirstTimestamp:2026-07-29 22:46:16.707850548 +0000 UTC m=+0.377967768,LastTimestamp:2026-07-29 22:46:16.707850548 +0000 UTC m=+0.377967768,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-242.ec2.internal,}"
Jul 29 22:46:16.727821 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.727806    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rn5ng"
Jul 29 22:46:16.728996 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:16.728942    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-242.ec2.internal.18c6e42b4006451f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-242.ec2.internal,UID:ip-10-0-128-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-242.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-242.ec2.internal,},FirstTimestamp:2026-07-29 22:46:16.707859743 +0000 UTC m=+0.377976963,LastTimestamp:2026-07-29 22:46:16.707859743 +0000 UTC m=+0.377976963,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-242.ec2.internal,}"
Jul 29 22:46:16.756182 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.756068    2581 manager.go:337] "Starting Device Plugin manager"
Jul 29 22:46:16.756249 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:16.756196    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 22:46:16.756249 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.756206    2581 server.go:87] "Starting device plugin registration server"
Jul 29 22:46:16.756388 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.756377    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 22:46:16.756440 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.756389    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 22:46:16.756498 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.756480    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 22:46:16.756573 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.756558    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 22:46:16.756573 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.756571    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 22:46:16.756974 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:16.756956    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 22:46:16.757052 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:16.757010    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-242.ec2.internal\" not found"
Jul 29 22:46:16.833002 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.832971    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 22:46:16.834073 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.834055    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 22:46:16.834136 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.834099    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 22:46:16.834136 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.834122    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 22:46:16.834251 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:16.834162    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 22:46:16.836714 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.836693    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 22:46:16.856728 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.856703    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:16.857433 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.857419    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:16.857496 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.857444    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:16.857496 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.857455    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:16.857496 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.857475    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-242.ec2.internal"
Jul 29 22:46:16.866259 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:16.866226    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-242.ec2.internal"
Jul 29 22:46:16.866259 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:16.866245    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-242.ec2.internal\": node \"ip-10-0-128-242.ec2.internal\" not found"
Jul 29 22:46:18.003693 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:18.003678    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-242.ec2.internal\" not found"
Jul 29 22:46:18.046268 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.046240    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal"]
Jul 29 22:46:18.046344 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.046311    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:18.047467 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.047453    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:18.047555 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.047482    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:18.047555 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.047496    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:18.048636 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.048620    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:18.048751 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.048736    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal"
Jul 29 22:46:18.048793 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.048764    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:18.049278 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.049262    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:18.049361 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.049291    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:18.049361 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.049305    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:18.049361 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.049317    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:18.049361 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.049340    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:18.049361 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.049353    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:18.050465 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.050450    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal"
Jul 29 22:46:18.050542 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.050487    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:18.051097 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.051084    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:18.051168 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.051108    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:18.051168 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.051117    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:18.074437 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:18.074414    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-242.ec2.internal\" not found" node="ip-10-0-128-242.ec2.internal"
Jul 29 22:46:18.078819 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:18.078791    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-242.ec2.internal\" not found" node="ip-10-0-128-242.ec2.internal"
Jul 29 22:46:18.103137 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.103117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/82d8343817331f460e5ea044619fe935-config\") pod \"kube-apiserver-proxy-ip-10-0-128-242.ec2.internal\" (UID: \"82d8343817331f460e5ea044619fe935\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal"
Jul 29 22:46:18.103203 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.103139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ceace035349ec6ca973ac8479406fd4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal\" (UID: \"6ceace035349ec6ca973ac8479406fd4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal"
Jul 29 22:46:18.103203 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.103156    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ceace035349ec6ca973ac8479406fd4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal\" (UID: \"6ceace035349ec6ca973ac8479406fd4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal"
Jul 29 22:46:18.104159 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:18.104145    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-242.ec2.internal\" not found"
Jul 29 22:46:18.203325 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.203306    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/82d8343817331f460e5ea044619fe935-config\") pod \"kube-apiserver-proxy-ip-10-0-128-242.ec2.internal\" (UID: \"82d8343817331f460e5ea044619fe935\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal"
Jul 29 22:46:18.203405 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.203330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/82d8343817331f460e5ea044619fe935-config\") pod \"kube-apiserver-proxy-ip-10-0-128-242.ec2.internal\" (UID: \"82d8343817331f460e5ea044619fe935\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal"
Jul 29 22:46:18.203405 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.203344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ceace035349ec6ca973ac8479406fd4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal\" (UID: \"6ceace035349ec6ca973ac8479406fd4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal"
Jul 29 22:46:18.203405 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.203362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ceace035349ec6ca973ac8479406fd4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal\" (UID: \"6ceace035349ec6ca973ac8479406fd4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal"
Jul 29 22:46:18.203405 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.203385    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ceace035349ec6ca973ac8479406fd4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal\" (UID: \"6ceace035349ec6ca973ac8479406fd4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal"
Jul 29 22:46:18.203527 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.203407    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ceace035349ec6ca973ac8479406fd4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal\" (UID: \"6ceace035349ec6ca973ac8479406fd4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal"
Jul 29 22:46:18.204385 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:18.204360    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-242.ec2.internal\" not found"
Jul 29 22:46:18.305098 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:18.305024    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-242.ec2.internal\" not found"
Jul 29 22:46:18.377256 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.377227    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal"
Jul 29 22:46:18.382054 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.382040    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal"
Jul 29 22:46:18.405122 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:18.405098    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-242.ec2.internal\" not found"
Jul 29 22:46:18.505630 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:18.505604    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-242.ec2.internal\" not found"
Jul 29 22:46:18.606073 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:18.606022    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-242.ec2.internal\" not found"
Jul 29 22:46:18.706632 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:18.706611    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-242.ec2.internal\" not found"
Jul 29 22:46:18.748784 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.748762    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 22:46:18.749201 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.748909    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 22:46:18.806820 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:18.806788    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-242.ec2.internal\" not found"
Jul 29 22:46:18.843617 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.843590    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 22:41:17 +0000 UTC" deadline="2028-05-01 15:15:34.232538146 +0000 UTC"
Jul 29 22:46:18.843690 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.843616    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15400h29m15.38892468s"
Jul 29 22:46:18.851674 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:18.851646    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ceace035349ec6ca973ac8479406fd4.slice/crio-8b46b61ff86718e9ae15b0ad5b2c32849200b1be6740e361a35cd40431a8aab5 WatchSource:0}: Error finding container 8b46b61ff86718e9ae15b0ad5b2c32849200b1be6740e361a35cd40431a8aab5: Status 404 returned error can't find the container with id 8b46b61ff86718e9ae15b0ad5b2c32849200b1be6740e361a35cd40431a8aab5
Jul 29 22:46:18.851945 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:18.851926    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d8343817331f460e5ea044619fe935.slice/crio-1d200caafac6e45f1b7490f31d2842ad6d0e025118d73aeb939643c3fd65c674 WatchSource:0}: Error finding container 1d200caafac6e45f1b7490f31d2842ad6d0e025118d73aeb939643c3fd65c674: Status 404 returned error can't find the container with id 1d200caafac6e45f1b7490f31d2842ad6d0e025118d73aeb939643c3fd65c674
Jul 29 22:46:18.858050 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.858034    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 22:46:18.907156 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:18.907134    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-242.ec2.internal\" not found"
Jul 29 22:46:18.949207 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.949170    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal" event={"ID":"6ceace035349ec6ca973ac8479406fd4","Type":"ContainerStarted","Data":"8b46b61ff86718e9ae15b0ad5b2c32849200b1be6740e361a35cd40431a8aab5"}
Jul 29 22:46:18.950260 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.950239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal" event={"ID":"82d8343817331f460e5ea044619fe935","Type":"ContainerStarted","Data":"1d200caafac6e45f1b7490f31d2842ad6d0e025118d73aeb939643c3fd65c674"}
Jul 29 22:46:18.986139 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:18.986119    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 22:46:19.000261 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.000248    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:19.001927 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.001911    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:19.001989 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.001939    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:19.001989 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.001949    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:19.007236 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:19.007223    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-242.ec2.internal\" not found"
Jul 29 22:46:19.108200 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:19.108157    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-242.ec2.internal\" not found"
Jul 29 22:46:19.115138 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.115119    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 22:46:19.201536 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.201513    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal"
Jul 29 22:46:19.215693 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.215674    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 22:46:19.217580 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.217567    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal"
Jul 29 22:46:19.224705 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.224691    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 22:46:19.375601 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.375527    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 22:46:19.775085 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.775021    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 22:46:19.781515 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.781494    2581 apiserver.go:52] "Watching apiserver"
Jul 29 22:46:19.790140 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.790118    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 22:46:19.790589 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.790566    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-7pv5p","kube-system/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq","openshift-cluster-node-tuning-operator/tuned-65x8z","openshift-image-registry/node-ca-8tftg","openshift-multus/multus-9cmk4","openshift-network-diagnostics/network-check-target-82qzb","kube-system/konnectivity-agent-6dn6d","openshift-dns/node-resolver-whv2s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal","openshift-multus/multus-additional-cni-plugins-z5plq","openshift-multus/network-metrics-daemon-2ksj4","openshift-network-operator/iptables-alerter-c4d6f","openshift-ovn-kubernetes/ovnkube-node-gzpfl"]
Jul 29 22:46:19.792023 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.792004    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.793202 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.793182    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.794165 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.794146    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:19.794333 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:19.794214    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82qzb" podUID="71e888ac-8fe5-4706-8af5-0c3227647e96"
Jul 29 22:46:19.794526 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.794501    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 22:46:19.794629 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.794564    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xltbk\""
Jul 29 22:46:19.795101 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.794914    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 22:46:19.795101 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.794914    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 22:46:19.795748 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.795700    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.795938 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.795854    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 22:46:19.796556 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.796201    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7fwlw\""
Jul 29 22:46:19.796556 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.796207    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 22:46:19.796556 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.796316    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 22:46:19.796773 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.796753    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 22:46:19.797008 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.796992    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8tftg"
Jul 29 22:46:19.798088 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.797923    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:19.798088 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:19.797993    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7pv5p" podUID="8687fc04-c1c6-4547-9787-dc94bfc700c3"
Jul 29 22:46:19.798305 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.798287    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qz2v5\""
Jul 29 22:46:19.798629 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.798475    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 22:46:19.798629 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.798568    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 22:46:19.799874 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.799854    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t55jd\""
Jul 29 22:46:19.800163 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.800144    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 22:46:19.801381 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.800916    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6dn6d"
Jul 29 22:46:19.801381 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.801071    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 22:46:19.801682 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.801626    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 22:46:19.803636 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.802709    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-whv2s"
Jul 29 22:46:19.803739 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.803701    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 22:46:19.804178 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.804046    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 22:46:19.804272 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.804244    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vq7gx\""
Jul 29 22:46:19.804414 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.804394    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.804974 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.804658    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 22:46:19.805656 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.805620    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 22:46:19.805656 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.805623    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5l9vz\""
Jul 29 22:46:19.806041 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.805892    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 22:46:19.806578 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.806496    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:19.806578 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:19.806570    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:46:19.807027 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.807009    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 22:46:19.807233 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.807216    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 22:46:19.807546 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.807514    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jhgqg\""
Jul 29 22:46:19.807948 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.807882    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c4d6f"
Jul 29 22:46:19.809230 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.809211    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.810716 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.810693    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 22:46:19.810848 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.810794    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 22:46:19.811042 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-etc-systemd\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.811128 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811053    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cwhnn\""
Jul 29 22:46:19.811128 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811056    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.811128 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811083    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-multus-cni-dir\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.811128 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-etc-sysctl-conf\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.811128 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811109    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 22:46:19.811128 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb349f88-27d7-4900-9d7c-d0e7916098ab-host\") pod \"node-ca-8tftg\" (UID: \"bb349f88-27d7-4900-9d7c-d0e7916098ab\") " pod="openshift-image-registry/node-ca-8tftg"
Jul 29 22:46:19.811407 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bb349f88-27d7-4900-9d7c-d0e7916098ab-serviceca\") pod \"node-ca-8tftg\" (UID: \"bb349f88-27d7-4900-9d7c-d0e7916098ab\") " pod="openshift-image-registry/node-ca-8tftg"
Jul 29 22:46:19.811407 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret\") pod \"global-pull-secret-syncer-7pv5p\" (UID: \"8687fc04-c1c6-4547-9787-dc94bfc700c3\") " pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:19.811407 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811193    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-system-cni-dir\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.811407 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811217    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/85bca852-e00d-4a1a-a746-11bd4cd94508-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.811407 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811272    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/85bca852-e00d-4a1a-a746-11bd4cd94508-sys-fs\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.811407 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-etc-kubernetes\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.811407 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8687fc04-c1c6-4547-9787-dc94bfc700c3-dbus\") pod \"global-pull-secret-syncer-7pv5p\" (UID: \"8687fc04-c1c6-4547-9787-dc94bfc700c3\") " pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:19.811407 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/85bca852-e00d-4a1a-a746-11bd4cd94508-etc-selinux\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.811766 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grvp2\" (UniqueName: \"kubernetes.io/projected/85bca852-e00d-4a1a-a746-11bd4cd94508-kube-api-access-grvp2\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.811766 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811438    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-cni-binary-copy\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.811766 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811488    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-host-run-k8s-cni-cncf-io\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.811766 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/950bb527-68a7-42b6-8aa4-b5242645d169-etc-tuned\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.811766 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811535    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/950bb527-68a7-42b6-8aa4-b5242645d169-tmp\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.811766 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-etc-modprobe-d\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.811766 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.811766 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/85bca852-e00d-4a1a-a746-11bd4cd94508-socket-dir\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.811766 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811628    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-etc-sysctl-d\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.811766 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811657    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a191d3f-9334-4881-8380-654a3b33d7d0-agent-certs\") pod \"konnectivity-agent-6dn6d\" (UID: \"9a191d3f-9334-4881-8380-654a3b33d7d0\") " pod="kube-system/konnectivity-agent-6dn6d"
Jul 29 22:46:19.811766 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqt4t\" (UniqueName: \"kubernetes.io/projected/bb349f88-27d7-4900-9d7c-d0e7916098ab-kube-api-access-fqt4t\") pod \"node-ca-8tftg\" (UID: \"bb349f88-27d7-4900-9d7c-d0e7916098ab\") " pod="openshift-image-registry/node-ca-8tftg"
Jul 29 22:46:19.811766 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811700    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-host-run-netns\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.811766 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811730    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxdzr\" (UniqueName: \"kubernetes.io/projected/0ad705ff-e172-497a-a972-108f30e7be6b-kube-api-access-mxdzr\") pod \"node-resolver-whv2s\" (UID: \"0ad705ff-e172-497a-a972-108f30e7be6b\") " pod="openshift-dns/node-resolver-whv2s"
Jul 29 22:46:19.811766 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811753    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-cnibin\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-etc-kubernetes\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-sys\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811941    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8687fc04-c1c6-4547-9787-dc94bfc700c3-kubelet-config\") pod \"global-pull-secret-syncer-7pv5p\" (UID: \"8687fc04-c1c6-4547-9787-dc94bfc700c3\") " pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.811961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a191d3f-9334-4881-8380-654a3b33d7d0-konnectivity-ca\") pod \"konnectivity-agent-6dn6d\" (UID: \"9a191d3f-9334-4881-8380-654a3b33d7d0\") " pod="kube-system/konnectivity-agent-6dn6d"
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ad705ff-e172-497a-a972-108f30e7be6b-hosts-file\") pod \"node-resolver-whv2s\" (UID: \"0ad705ff-e172-497a-a972-108f30e7be6b\") " pod="openshift-dns/node-resolver-whv2s"
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-os-release\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812080    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812102    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-lib-modules\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812158    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ad705ff-e172-497a-a972-108f30e7be6b-tmp-dir\") pod \"node-resolver-whv2s\" (UID: \"0ad705ff-e172-497a-a972-108f30e7be6b\") " pod="openshift-dns/node-resolver-whv2s"
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812183    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlrnq\" (UniqueName: \"kubernetes.io/projected/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-kube-api-access-tlrnq\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812209    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-system-cni-dir\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812235    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-multus-socket-dir-parent\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812259    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-host-var-lib-cni-bin\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-etc-sysconfig\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812293    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812322    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-cni-binary-copy\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812333    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 22:46:19.812434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812341    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m64b9\""
Jul 29 22:46:19.813296 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.813296 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812377    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh7v2\" (UniqueName: \"kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2\") pod \"network-check-target-82qzb\" (UID: \"71e888ac-8fe5-4706-8af5-0c3227647e96\") " pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:19.813296 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-cnibin\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.813296 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812433    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-run\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.813296 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812457    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8blc5\" (UniqueName: \"kubernetes.io/projected/950bb527-68a7-42b6-8aa4-b5242645d169-kube-api-access-8blc5\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.813296 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-var-lib-kubelet\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.813296 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-os-release\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.813296 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-multus-daemon-config\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.813296 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-host\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.813296 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-multus-conf-dir\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.813296 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-host-run-multus-certs\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.813296 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812627    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7d9g\" (UniqueName: \"kubernetes.io/projected/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-kube-api-access-k7d9g\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.813296 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/85bca852-e00d-4a1a-a746-11bd4cd94508-registration-dir\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.813296 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812683    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/85bca852-e00d-4a1a-a746-11bd4cd94508-device-dir\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.813296 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-host-var-lib-cni-multus\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.813296 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812736    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-host-var-lib-kubelet\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.813296 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.812760    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-hostroot\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.814136 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.813339    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 22:46:19.814136 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.813573    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 22:46:19.816444 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.816333    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 22:46:19.842031 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.842009    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b8dqg"
Jul 29 22:46:19.849483 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.849467    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b8dqg"
Jul 29 22:46:19.902971 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.902950    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 22:46:19.913239 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-host-run-multus-certs\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.913352 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7d9g\" (UniqueName: \"kubernetes.io/projected/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-kube-api-access-k7d9g\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.913352 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913280    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-host-run-multus-certs\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.913352 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-etc-openvswitch\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.913512 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6166a6a8-c059-4854-a4c7-51904b3265d6-env-overrides\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.913512 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/85bca852-e00d-4a1a-a746-11bd4cd94508-registration-dir\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.913512 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/85bca852-e00d-4a1a-a746-11bd4cd94508-device-dir\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.913512 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913484    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/85bca852-e00d-4a1a-a746-11bd4cd94508-registration-dir\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.913693 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-host-var-lib-cni-multus\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.913693 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-host-var-lib-kubelet\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.913693 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-hostroot\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.913693 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913617    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-host-var-lib-cni-multus\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.913693 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-host-var-lib-kubelet\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.913693 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913537    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/85bca852-e00d-4a1a-a746-11bd4cd94508-device-dir\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.913693 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913621    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-kubelet\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.913693 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-hostroot\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.913693 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs\") pod \"network-metrics-daemon-2ksj4\" (UID: \"cf450310-9f32-4f65-96ea-b4d676ff2df7\") " pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:19.914042 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913715    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ds9c\" (UniqueName: \"kubernetes.io/projected/cf450310-9f32-4f65-96ea-b4d676ff2df7-kube-api-access-6ds9c\") pod \"network-metrics-daemon-2ksj4\" (UID: \"cf450310-9f32-4f65-96ea-b4d676ff2df7\") " pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:19.914042 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-etc-systemd\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.914042 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.914042 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-multus-cni-dir\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.914042 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-node-log\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.914310 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.913828    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-etc-systemd\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.914310 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-cni-bin\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.914310 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914227    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/556818de-e974-4303-80b3-846815ab4cd3-host-slash\") pod \"iptables-alerter-c4d6f\" (UID: \"556818de-e974-4303-80b3-846815ab4cd3\") " pod="openshift-network-operator/iptables-alerter-c4d6f"
Jul 29 22:46:19.914310 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-etc-sysctl-conf\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.914310 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-multus-cni-dir\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.914530 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb349f88-27d7-4900-9d7c-d0e7916098ab-host\") pod \"node-ca-8tftg\" (UID: \"bb349f88-27d7-4900-9d7c-d0e7916098ab\") " pod="openshift-image-registry/node-ca-8tftg"
Jul 29 22:46:19.914530 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bb349f88-27d7-4900-9d7c-d0e7916098ab-serviceca\") pod \"node-ca-8tftg\" (UID: \"bb349f88-27d7-4900-9d7c-d0e7916098ab\") " pod="openshift-image-registry/node-ca-8tftg"
Jul 29 22:46:19.914530 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret\") pod \"global-pull-secret-syncer-7pv5p\" (UID: \"8687fc04-c1c6-4547-9787-dc94bfc700c3\") " pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:19.914530 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6166a6a8-c059-4854-a4c7-51904b3265d6-ovnkube-script-lib\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.914530 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-etc-sysctl-conf\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.914530 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-system-cni-dir\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.914530 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/85bca852-e00d-4a1a-a746-11bd4cd94508-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.914883 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/85bca852-e00d-4a1a-a746-11bd4cd94508-sys-fs\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.914883 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914608    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-etc-kubernetes\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.914883 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:19.914653    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:19.914883 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914731    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/85bca852-e00d-4a1a-a746-11bd4cd94508-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.914883 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:19.914766    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret podName:8687fc04-c1c6-4547-9787-dc94bfc700c3 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:20.414745169 +0000 UTC m=+2.973089979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret") pod "global-pull-secret-syncer-7pv5p" (UID: "8687fc04-c1c6-4547-9787-dc94bfc700c3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:19.914883 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/85bca852-e00d-4a1a-a746-11bd4cd94508-sys-fs\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.914883 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-system-cni-dir\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.915215 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-etc-kubernetes\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.915215 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914922    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bb349f88-27d7-4900-9d7c-d0e7916098ab-serviceca\") pod \"node-ca-8tftg\" (UID: \"bb349f88-27d7-4900-9d7c-d0e7916098ab\") " pod="openshift-image-registry/node-ca-8tftg"
Jul 29 22:46:19.915215 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914958    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb349f88-27d7-4900-9d7c-d0e7916098ab-host\") pod \"node-ca-8tftg\" (UID: \"bb349f88-27d7-4900-9d7c-d0e7916098ab\") " pod="openshift-image-registry/node-ca-8tftg"
Jul 29 22:46:19.915215 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.914991    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.915215 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915102    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-run-netns\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.915215 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915153    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/556818de-e974-4303-80b3-846815ab4cd3-iptables-alerter-script\") pod \"iptables-alerter-c4d6f\" (UID: \"556818de-e974-4303-80b3-846815ab4cd3\") " pod="openshift-network-operator/iptables-alerter-c4d6f"
Jul 29 22:46:19.915215 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkvp4\" (UniqueName: \"kubernetes.io/projected/556818de-e974-4303-80b3-846815ab4cd3-kube-api-access-mkvp4\") pod \"iptables-alerter-c4d6f\" (UID: \"556818de-e974-4303-80b3-846815ab4cd3\") " pod="openshift-network-operator/iptables-alerter-c4d6f"
Jul 29 22:46:19.915511 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8687fc04-c1c6-4547-9787-dc94bfc700c3-dbus\") pod \"global-pull-secret-syncer-7pv5p\" (UID: \"8687fc04-c1c6-4547-9787-dc94bfc700c3\") " pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:19.915511 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/85bca852-e00d-4a1a-a746-11bd4cd94508-etc-selinux\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.915511 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8687fc04-c1c6-4547-9787-dc94bfc700c3-dbus\") pod \"global-pull-secret-syncer-7pv5p\" (UID: \"8687fc04-c1c6-4547-9787-dc94bfc700c3\") " pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:19.915511 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grvp2\" (UniqueName: \"kubernetes.io/projected/85bca852-e00d-4a1a-a746-11bd4cd94508-kube-api-access-grvp2\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.915511 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-cni-binary-copy\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.915511 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-host-run-k8s-cni-cncf-io\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.915511 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915476    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/85bca852-e00d-4a1a-a746-11bd4cd94508-etc-selinux\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.915511 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-systemd-units\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.915902 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/950bb527-68a7-42b6-8aa4-b5242645d169-etc-tuned\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.915902 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/950bb527-68a7-42b6-8aa4-b5242645d169-tmp\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.915902 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.915902 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-etc-modprobe-d\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.915902 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.915902 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/85bca852-e00d-4a1a-a746-11bd4cd94508-socket-dir\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.915902 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915767    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-slash\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.915902 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.915794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6166a6a8-c059-4854-a4c7-51904b3265d6-ovn-node-metrics-cert\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.916318 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.916298    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 22:46:19.916410 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.916393    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-cni-binary-copy\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.916499 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.916476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-etc-sysctl-d\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.916549 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.916515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a191d3f-9334-4881-8380-654a3b33d7d0-agent-certs\") pod \"konnectivity-agent-6dn6d\" (UID: \"9a191d3f-9334-4881-8380-654a3b33d7d0\") " pod="kube-system/konnectivity-agent-6dn6d"
Jul 29 22:46:19.916592 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.916540    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqt4t\" (UniqueName: \"kubernetes.io/projected/bb349f88-27d7-4900-9d7c-d0e7916098ab-kube-api-access-fqt4t\") pod \"node-ca-8tftg\" (UID: \"bb349f88-27d7-4900-9d7c-d0e7916098ab\") " pod="openshift-image-registry/node-ca-8tftg"
Jul 29 22:46:19.916592 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.916585    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-host-run-netns\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.916683 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.916615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6166a6a8-c059-4854-a4c7-51904b3265d6-ovnkube-config\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.916683 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.916646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxdzr\" (UniqueName: \"kubernetes.io/projected/0ad705ff-e172-497a-a972-108f30e7be6b-kube-api-access-mxdzr\") pod \"node-resolver-whv2s\" (UID: \"0ad705ff-e172-497a-a972-108f30e7be6b\") " pod="openshift-dns/node-resolver-whv2s"
Jul 29 22:46:19.916771 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.916721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-cnibin\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.916771 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.916751    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-run-systemd\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.916881 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.916774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-etc-kubernetes\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.916946 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.916907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-sys\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.916997 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.916949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8687fc04-c1c6-4547-9787-dc94bfc700c3-kubelet-config\") pod \"global-pull-secret-syncer-7pv5p\" (UID: \"8687fc04-c1c6-4547-9787-dc94bfc700c3\") " pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:19.917040 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.916991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-var-lib-openvswitch\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.917040 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.917032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-cni-netd\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.917130 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.917071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a191d3f-9334-4881-8380-654a3b33d7d0-konnectivity-ca\") pod \"konnectivity-agent-6dn6d\" (UID: \"9a191d3f-9334-4881-8380-654a3b33d7d0\") " pod="kube-system/konnectivity-agent-6dn6d"
Jul 29 22:46:19.917130 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.917099    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ad705ff-e172-497a-a972-108f30e7be6b-hosts-file\") pod \"node-resolver-whv2s\" (UID: \"0ad705ff-e172-497a-a972-108f30e7be6b\") " pod="openshift-dns/node-resolver-whv2s"
Jul 29 22:46:19.917247 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.917169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-os-release\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.917247 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.917197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-lib-modules\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.917343 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.917247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ad705ff-e172-497a-a972-108f30e7be6b-tmp-dir\") pod \"node-resolver-whv2s\" (UID: \"0ad705ff-e172-497a-a972-108f30e7be6b\") " pod="openshift-dns/node-resolver-whv2s"
Jul 29 22:46:19.917343 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.917276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlrnq\" (UniqueName: \"kubernetes.io/projected/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-kube-api-access-tlrnq\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.917343 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.917327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-system-cni-dir\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.917471 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.917354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-multus-socket-dir-parent\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.917471 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.917381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-host-var-lib-cni-bin\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.917471 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.917417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-run-ovn\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.917607 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.917497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-etc-sysconfig\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.917607 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.917522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-cni-binary-copy\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.917693 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.917663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.917739 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.917692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7v2\" (UniqueName: \"kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2\") pod \"network-check-target-82qzb\" (UID: \"71e888ac-8fe5-4706-8af5-0c3227647e96\") " pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:19.917739 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.917719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-cnibin\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.917912 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.917843    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.918036 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.918002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-system-cni-dir\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.918101 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.918071    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-lib-modules\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.918155 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.918100    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.918155 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.918097    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-host-var-lib-cni-bin\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.918814 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.918335    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-multus-socket-dir-parent\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.918814 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.918472    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ad705ff-e172-497a-a972-108f30e7be6b-tmp-dir\") pod \"node-resolver-whv2s\" (UID: \"0ad705ff-e172-497a-a972-108f30e7be6b\") " pod="openshift-dns/node-resolver-whv2s"
Jul 29 22:46:19.918814 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.918682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-etc-sysctl-d\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.918814 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.918707    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-host-run-netns\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.919126 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.918997    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.919183 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.919157    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-cni-binary-copy\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.919869 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.919741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbx79\" (UniqueName: \"kubernetes.io/projected/6166a6a8-c059-4854-a4c7-51904b3265d6-kube-api-access-rbx79\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.921758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.920554    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-etc-sysconfig\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.921758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.920627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-sys\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.921758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.920716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-etc-modprobe-d\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.921758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.920764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8687fc04-c1c6-4547-9787-dc94bfc700c3-kubelet-config\") pod \"global-pull-secret-syncer-7pv5p\" (UID: \"8687fc04-c1c6-4547-9787-dc94bfc700c3\") " pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:19.921758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.920776    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/950bb527-68a7-42b6-8aa4-b5242645d169-etc-tuned\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.921758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.920813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-run\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.921758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.920850    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-run\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.921758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.920882    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8blc5\" (UniqueName: \"kubernetes.io/projected/950bb527-68a7-42b6-8aa4-b5242645d169-kube-api-access-8blc5\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.921758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.920915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-run-openvswitch\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.921758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.920958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-log-socket\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:19.921758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-var-lib-kubelet\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.921758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921076    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-os-release\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.921758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-multus-daemon-config\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.921758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-host\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.921758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921155    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-multus-conf-dir\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.921758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921156    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-var-lib-kubelet\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.921758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921215    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-os-release\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.921758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921240    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-multus-conf-dir\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.922656 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-cnibin\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.922656 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-cnibin\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.922656 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/85bca852-e00d-4a1a-a746-11bd4cd94508-socket-dir\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.922656 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921437    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a191d3f-9334-4881-8380-654a3b33d7d0-agent-certs\") pod \"konnectivity-agent-6dn6d\" (UID: \"9a191d3f-9334-4881-8380-654a3b33d7d0\") " pod="kube-system/konnectivity-agent-6dn6d"
Jul 29 22:46:19.922656 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921450    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-host-run-k8s-cni-cncf-io\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.922656 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ad705ff-e172-497a-a972-108f30e7be6b-hosts-file\") pod \"node-resolver-whv2s\" (UID: \"0ad705ff-e172-497a-a972-108f30e7be6b\") " pod="openshift-dns/node-resolver-whv2s"
Jul 29 22:46:19.922656 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921510    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-os-release\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.922656 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-host\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.922656 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921677    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-multus-daemon-config\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.922656 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921720    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a191d3f-9334-4881-8380-654a3b33d7d0-konnectivity-ca\") pod \"konnectivity-agent-6dn6d\" (UID: \"9a191d3f-9334-4881-8380-654a3b33d7d0\") " pod="kube-system/konnectivity-agent-6dn6d"
Jul 29 22:46:19.922656 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.921727    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/950bb527-68a7-42b6-8aa4-b5242645d169-etc-kubernetes\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.922656 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.922157    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7d9g\" (UniqueName: \"kubernetes.io/projected/19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96-kube-api-access-k7d9g\") pod \"multus-9cmk4\" (UID: \"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96\") " pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:19.923890 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.923829    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grvp2\" (UniqueName: \"kubernetes.io/projected/85bca852-e00d-4a1a-a746-11bd4cd94508-kube-api-access-grvp2\") pod \"aws-ebs-csi-driver-node-rhqrq\" (UID: \"85bca852-e00d-4a1a-a746-11bd4cd94508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:19.924110 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.924023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/950bb527-68a7-42b6-8aa4-b5242645d169-tmp\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.927170 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:19.927150    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:19.927254 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:19.927173    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:19.927254 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:19.927186    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vh7v2 for pod openshift-network-diagnostics/network-check-target-82qzb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:19.927353 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:19.927261    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2 podName:71e888ac-8fe5-4706-8af5-0c3227647e96 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:20.427229653 +0000 UTC m=+2.985574464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vh7v2" (UniqueName: "kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2") pod "network-check-target-82qzb" (UID: "71e888ac-8fe5-4706-8af5-0c3227647e96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:19.929730 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.929506    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8blc5\" (UniqueName: \"kubernetes.io/projected/950bb527-68a7-42b6-8aa4-b5242645d169-kube-api-access-8blc5\") pod \"tuned-65x8z\" (UID: \"950bb527-68a7-42b6-8aa4-b5242645d169\") " pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:19.929730 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.929690    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlrnq\" (UniqueName: \"kubernetes.io/projected/0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8-kube-api-access-tlrnq\") pod \"multus-additional-cni-plugins-z5plq\" (UID: \"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8\") " pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:19.929961 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.929934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqt4t\" (UniqueName: \"kubernetes.io/projected/bb349f88-27d7-4900-9d7c-d0e7916098ab-kube-api-access-fqt4t\") pod \"node-ca-8tftg\" (UID: \"bb349f88-27d7-4900-9d7c-d0e7916098ab\") " pod="openshift-image-registry/node-ca-8tftg"
Jul 29 22:46:19.930037 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:19.929931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxdzr\" (UniqueName: \"kubernetes.io/projected/0ad705ff-e172-497a-a972-108f30e7be6b-kube-api-access-mxdzr\") pod \"node-resolver-whv2s\" (UID: \"0ad705ff-e172-497a-a972-108f30e7be6b\") " pod="openshift-dns/node-resolver-whv2s"
Jul 29 22:46:20.021470 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.021440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6166a6a8-c059-4854-a4c7-51904b3265d6-ovnkube-script-lib\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.021616 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.021481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-run-netns\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.021616 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.021503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/556818de-e974-4303-80b3-846815ab4cd3-iptables-alerter-script\") pod \"iptables-alerter-c4d6f\" (UID: \"556818de-e974-4303-80b3-846815ab4cd3\") " pod="openshift-network-operator/iptables-alerter-c4d6f"
Jul 29 22:46:20.021616 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.021528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkvp4\" (UniqueName: \"kubernetes.io/projected/556818de-e974-4303-80b3-846815ab4cd3-kube-api-access-mkvp4\") pod \"iptables-alerter-c4d6f\" (UID: \"556818de-e974-4303-80b3-846815ab4cd3\") " pod="openshift-network-operator/iptables-alerter-c4d6f"
Jul 29 22:46:20.021616 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.021570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-systemd-units\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.021616 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.021603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.021854 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.021639    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-slash\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.021854 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.021663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6166a6a8-c059-4854-a4c7-51904b3265d6-ovn-node-metrics-cert\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.021854 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.021789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6166a6a8-c059-4854-a4c7-51904b3265d6-ovnkube-config\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.021987 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.021851    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-run-netns\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.021987 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.021843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-run-systemd\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.021987 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.021898    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-run-systemd\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.021987 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.021907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-var-lib-openvswitch\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.021987 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.021937    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-cni-netd\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022263 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-run-ovn\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022263 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022024    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022263 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022263 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbx79\" (UniqueName: \"kubernetes.io/projected/6166a6a8-c059-4854-a4c7-51904b3265d6-kube-api-access-rbx79\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022263 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-systemd-units\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022263 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-run-openvswitch\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022263 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-log-socket\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022263 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022130    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-cni-netd\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022263 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-etc-openvswitch\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022263 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022176    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-log-socket\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022263 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6166a6a8-c059-4854-a4c7-51904b3265d6-env-overrides\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022263 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022024    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-slash\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022263 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022255    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-var-lib-openvswitch\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022299    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022282    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-run-ovn\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-etc-openvswitch\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-kubelet\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022338    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-run-openvswitch\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022344    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/556818de-e974-4303-80b3-846815ab4cd3-iptables-alerter-script\") pod \"iptables-alerter-c4d6f\" (UID: \"556818de-e974-4303-80b3-846815ab4cd3\") " pod="openshift-network-operator/iptables-alerter-c4d6f"
Jul 29 22:46:20.022916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022443    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-kubelet\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs\") pod \"network-metrics-daemon-2ksj4\" (UID: \"cf450310-9f32-4f65-96ea-b4d676ff2df7\") " pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:20.022916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022531    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ds9c\" (UniqueName: \"kubernetes.io/projected/cf450310-9f32-4f65-96ea-b4d676ff2df7-kube-api-access-6ds9c\") pod \"network-metrics-daemon-2ksj4\" (UID: \"cf450310-9f32-4f65-96ea-b4d676ff2df7\") " pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:20.022916 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:20.022577    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:20.022916 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:20.022626    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs podName:cf450310-9f32-4f65-96ea-b4d676ff2df7 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:20.522609623 +0000 UTC m=+3.080954431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs") pod "network-metrics-daemon-2ksj4" (UID: "cf450310-9f32-4f65-96ea-b4d676ff2df7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:20.022916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-node-log\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-cni-bin\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/556818de-e974-4303-80b3-846815ab4cd3-host-slash\") pod \"iptables-alerter-c4d6f\" (UID: \"556818de-e974-4303-80b3-846815ab4cd3\") " pod="openshift-network-operator/iptables-alerter-c4d6f"
Jul 29 22:46:20.022916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022779    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/556818de-e974-4303-80b3-846815ab4cd3-host-slash\") pod \"iptables-alerter-c4d6f\" (UID: \"556818de-e974-4303-80b3-846815ab4cd3\") " pod="openshift-network-operator/iptables-alerter-c4d6f"
Jul 29 22:46:20.022916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022821    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-node-log\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.022916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.022834    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6166a6a8-c059-4854-a4c7-51904b3265d6-host-cni-bin\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.023591 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.023021    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6166a6a8-c059-4854-a4c7-51904b3265d6-env-overrides\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.023591 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.023103    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6166a6a8-c059-4854-a4c7-51904b3265d6-ovnkube-config\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.023591 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.023273    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6166a6a8-c059-4854-a4c7-51904b3265d6-ovnkube-script-lib\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.025124 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.025070    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6166a6a8-c059-4854-a4c7-51904b3265d6-ovn-node-metrics-cert\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.030510 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.030488    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbx79\" (UniqueName: \"kubernetes.io/projected/6166a6a8-c059-4854-a4c7-51904b3265d6-kube-api-access-rbx79\") pod \"ovnkube-node-gzpfl\" (UID: \"6166a6a8-c059-4854-a4c7-51904b3265d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.033587 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.033567    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ds9c\" (UniqueName: \"kubernetes.io/projected/cf450310-9f32-4f65-96ea-b4d676ff2df7-kube-api-access-6ds9c\") pod \"network-metrics-daemon-2ksj4\" (UID: \"cf450310-9f32-4f65-96ea-b4d676ff2df7\") " pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:20.033738 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.033717    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkvp4\" (UniqueName: \"kubernetes.io/projected/556818de-e974-4303-80b3-846815ab4cd3-kube-api-access-mkvp4\") pod \"iptables-alerter-c4d6f\" (UID: \"556818de-e974-4303-80b3-846815ab4cd3\") " pod="openshift-network-operator/iptables-alerter-c4d6f"
Jul 29 22:46:20.106574 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.106545    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq"
Jul 29 22:46:20.115265 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.115245    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9cmk4"
Jul 29 22:46:20.122673 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.122654    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-65x8z"
Jul 29 22:46:20.129065 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.129048    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8tftg"
Jul 29 22:46:20.137340 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.137321    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6dn6d"
Jul 29 22:46:20.144675 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.144651    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-whv2s"
Jul 29 22:46:20.151937 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.151920    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z5plq"
Jul 29 22:46:20.160782 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.160762    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c4d6f"
Jul 29 22:46:20.164351 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.164334    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:20.426649 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.426567    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret\") pod \"global-pull-secret-syncer-7pv5p\" (UID: \"8687fc04-c1c6-4547-9787-dc94bfc700c3\") " pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:20.426793 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:20.426730    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:20.426879 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:20.426809    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret podName:8687fc04-c1c6-4547-9787-dc94bfc700c3 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:21.426781376 +0000 UTC m=+3.985126174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret") pod "global-pull-secret-syncer-7pv5p" (UID: "8687fc04-c1c6-4547-9787-dc94bfc700c3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:20.527820 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.527775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs\") pod \"network-metrics-daemon-2ksj4\" (UID: \"cf450310-9f32-4f65-96ea-b4d676ff2df7\") " pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:20.528021 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.527891    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7v2\" (UniqueName: \"kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2\") pod \"network-check-target-82qzb\" (UID: \"71e888ac-8fe5-4706-8af5-0c3227647e96\") " pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:20.528021 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:20.527910    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:20.528021 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:20.527994    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs podName:cf450310-9f32-4f65-96ea-b4d676ff2df7 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:21.527974529 +0000 UTC m=+4.086319326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs") pod "network-metrics-daemon-2ksj4" (UID: "cf450310-9f32-4f65-96ea-b4d676ff2df7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:20.528186 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:20.528056    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:20.528186 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:20.528077    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:20.528186 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:20.528088    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vh7v2 for pod openshift-network-diagnostics/network-check-target-82qzb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:20.528186 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:20.528138    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2 podName:71e888ac-8fe5-4706-8af5-0c3227647e96 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:21.528119713 +0000 UTC m=+4.086464519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vh7v2" (UniqueName: "kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2") pod "network-check-target-82qzb" (UID: "71e888ac-8fe5-4706-8af5-0c3227647e96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:20.706969 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:20.706838    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad705ff_e172_497a_a972_108f30e7be6b.slice/crio-ef4f776c97c3f38e4e76a00511ee69890e0ec57a01a7e3eaf52153e418ed5092 WatchSource:0}: Error finding container ef4f776c97c3f38e4e76a00511ee69890e0ec57a01a7e3eaf52153e418ed5092: Status 404 returned error can't find the container with id ef4f776c97c3f38e4e76a00511ee69890e0ec57a01a7e3eaf52153e418ed5092
Jul 29 22:46:20.708049 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:20.707898    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a191d3f_9334_4881_8380_654a3b33d7d0.slice/crio-e72e076c1a2b716e76532067c124925f505677baed61579ae6e5d0b75a484e15 WatchSource:0}: Error finding container e72e076c1a2b716e76532067c124925f505677baed61579ae6e5d0b75a484e15: Status 404 returned error can't find the container with id e72e076c1a2b716e76532067c124925f505677baed61579ae6e5d0b75a484e15
Jul 29 22:46:20.714461 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:20.710586    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb349f88_27d7_4900_9d7c_d0e7916098ab.slice/crio-dac32045abab1c6192c547715dd9a6b74c52da34adf1ed2c25d34b889c16ea20 WatchSource:0}: Error finding container dac32045abab1c6192c547715dd9a6b74c52da34adf1ed2c25d34b889c16ea20: Status 404 returned error can't find the container with id dac32045abab1c6192c547715dd9a6b74c52da34adf1ed2c25d34b889c16ea20
Jul 29 22:46:20.715384 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:20.715329    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2daf9f_e88e_4f93_a21a_f3a7bcc6a6b8.slice/crio-b199293d51e081b37ff86a97e64f9ad05bb61d245c1f91ed87def761731f24b1 WatchSource:0}: Error finding container b199293d51e081b37ff86a97e64f9ad05bb61d245c1f91ed87def761731f24b1: Status 404 returned error can't find the container with id b199293d51e081b37ff86a97e64f9ad05bb61d245c1f91ed87def761731f24b1
Jul 29 22:46:20.715983 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:20.715965    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556818de_e974_4303_80b3_846815ab4cd3.slice/crio-43721b346057d75c66e29fb744718ac53316aaa00c50a5694b9f6e18bead0bfd WatchSource:0}: Error finding container 43721b346057d75c66e29fb744718ac53316aaa00c50a5694b9f6e18bead0bfd: Status 404 returned error can't find the container with id 43721b346057d75c66e29fb744718ac53316aaa00c50a5694b9f6e18bead0bfd
Jul 29 22:46:20.717276 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:20.717206    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6166a6a8_c059_4854_a4c7_51904b3265d6.slice/crio-505b901f105ef64d7b276527670675f5663dc246a19227173790fc967e1f097b WatchSource:0}: Error finding container 505b901f105ef64d7b276527670675f5663dc246a19227173790fc967e1f097b: Status 404 returned error can't find the container with id 505b901f105ef64d7b276527670675f5663dc246a19227173790fc967e1f097b
Jul 29 22:46:20.718580 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:20.718555    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bca852_e00d_4a1a_a746_11bd4cd94508.slice/crio-25e0b82648b152e6385a1316df1775663e6e09b10ef8f09a64c9c7abe8db486e WatchSource:0}: Error finding container 25e0b82648b152e6385a1316df1775663e6e09b10ef8f09a64c9c7abe8db486e: Status 404 returned error can't find the container with id 25e0b82648b152e6385a1316df1775663e6e09b10ef8f09a64c9c7abe8db486e
Jul 29 22:46:20.719788 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:20.719765    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950bb527_68a7_42b6_8aa4_b5242645d169.slice/crio-b836bfdec045f97670b903dd289f5932953e94eb096e45cd214dca10b08300b6 WatchSource:0}: Error finding container b836bfdec045f97670b903dd289f5932953e94eb096e45cd214dca10b08300b6: Status 404 returned error can't find the container with id b836bfdec045f97670b903dd289f5932953e94eb096e45cd214dca10b08300b6
Jul 29 22:46:20.720919 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:20.720539    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19fe18f8_824f_42c1_ac3e_e3a1bd0e1b96.slice/crio-38e315c341283b8bf5f70aa50f4b3aeee5af9db68a20fca81d145575f23ecdc6 WatchSource:0}: Error finding container 38e315c341283b8bf5f70aa50f4b3aeee5af9db68a20fca81d145575f23ecdc6: Status 404 returned error can't find the container with id 38e315c341283b8bf5f70aa50f4b3aeee5af9db68a20fca81d145575f23ecdc6
Jul 29 22:46:20.851156 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.851102    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 22:41:19 +0000 UTC" deadline="2028-01-15 10:09:04.105657652 +0000 UTC"
Jul 29 22:46:20.851156 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.851153    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12827h22m43.254506883s"
Jul 29 22:46:20.947243 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.947087    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:20.947327 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:20.947276    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:46:20.954848 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.954824    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-65x8z" event={"ID":"950bb527-68a7-42b6-8aa4-b5242645d169","Type":"ContainerStarted","Data":"b836bfdec045f97670b903dd289f5932953e94eb096e45cd214dca10b08300b6"}
Jul 29 22:46:20.955837 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.955818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9cmk4" event={"ID":"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96","Type":"ContainerStarted","Data":"38e315c341283b8bf5f70aa50f4b3aeee5af9db68a20fca81d145575f23ecdc6"}
Jul 29 22:46:20.956701 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.956679    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq" event={"ID":"85bca852-e00d-4a1a-a746-11bd4cd94508","Type":"ContainerStarted","Data":"25e0b82648b152e6385a1316df1775663e6e09b10ef8f09a64c9c7abe8db486e"}
Jul 29 22:46:20.957564 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.957545    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8tftg" event={"ID":"bb349f88-27d7-4900-9d7c-d0e7916098ab","Type":"ContainerStarted","Data":"dac32045abab1c6192c547715dd9a6b74c52da34adf1ed2c25d34b889c16ea20"}
Jul 29 22:46:20.958543 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.958526    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6dn6d" event={"ID":"9a191d3f-9334-4881-8380-654a3b33d7d0","Type":"ContainerStarted","Data":"e72e076c1a2b716e76532067c124925f505677baed61579ae6e5d0b75a484e15"}
Jul 29 22:46:20.959436 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.959421    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-whv2s" event={"ID":"0ad705ff-e172-497a-a972-108f30e7be6b","Type":"ContainerStarted","Data":"ef4f776c97c3f38e4e76a00511ee69890e0ec57a01a7e3eaf52153e418ed5092"}
Jul 29 22:46:20.960378 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.960362    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c4d6f" event={"ID":"556818de-e974-4303-80b3-846815ab4cd3","Type":"ContainerStarted","Data":"43721b346057d75c66e29fb744718ac53316aaa00c50a5694b9f6e18bead0bfd"}
Jul 29 22:46:20.961400 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.961380    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl" event={"ID":"6166a6a8-c059-4854-a4c7-51904b3265d6","Type":"ContainerStarted","Data":"505b901f105ef64d7b276527670675f5663dc246a19227173790fc967e1f097b"}
Jul 29 22:46:20.962328 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.962311    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z5plq" event={"ID":"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8","Type":"ContainerStarted","Data":"b199293d51e081b37ff86a97e64f9ad05bb61d245c1f91ed87def761731f24b1"}
Jul 29 22:46:20.963945 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.963924    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ceace035349ec6ca973ac8479406fd4" containerID="6879b7065a486b0d3949d2aee90ab5ea561c1ce28624d8d0b1e8cea7045d25c0" exitCode=0
Jul 29 22:46:20.964026 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.963987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal" event={"ID":"6ceace035349ec6ca973ac8479406fd4","Type":"ContainerDied","Data":"6879b7065a486b0d3949d2aee90ab5ea561c1ce28624d8d0b1e8cea7045d25c0"}
Jul 29 22:46:20.965353 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.965336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal" event={"ID":"82d8343817331f460e5ea044619fe935","Type":"ContainerStarted","Data":"7375cb823e7321cdc766e228b131765b437b0668e541cdebb9a64c4ebd9773b0"}
Jul 29 22:46:20.995964 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:20.995911    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal" podStartSLOduration=1.995898221 podStartE2EDuration="1.995898221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:20.99573754 +0000 UTC m=+3.554082359" watchObservedRunningTime="2026-07-29 22:46:20.995898221 +0000 UTC m=+3.554243036"
Jul 29 22:46:21.434974 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:21.434890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret\") pod \"global-pull-secret-syncer-7pv5p\" (UID: \"8687fc04-c1c6-4547-9787-dc94bfc700c3\") " pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:21.435129 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:21.435040    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:21.435129 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:21.435102    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret podName:8687fc04-c1c6-4547-9787-dc94bfc700c3 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:23.435085056 +0000 UTC m=+5.993429867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret") pod "global-pull-secret-syncer-7pv5p" (UID: "8687fc04-c1c6-4547-9787-dc94bfc700c3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:21.536298 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:21.536155    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs\") pod \"network-metrics-daemon-2ksj4\" (UID: \"cf450310-9f32-4f65-96ea-b4d676ff2df7\") " pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:21.536298 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:21.536246    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7v2\" (UniqueName: \"kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2\") pod \"network-check-target-82qzb\" (UID: \"71e888ac-8fe5-4706-8af5-0c3227647e96\") " pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:21.536535 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:21.536320    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:21.536535 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:21.536361    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:21.536535 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:21.536380    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:21.536535 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:21.536391    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vh7v2 for pod openshift-network-diagnostics/network-check-target-82qzb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:21.536535 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:21.536389    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs podName:cf450310-9f32-4f65-96ea-b4d676ff2df7 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:23.536370099 +0000 UTC m=+6.094714916 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs") pod "network-metrics-daemon-2ksj4" (UID: "cf450310-9f32-4f65-96ea-b4d676ff2df7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:21.536535 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:21.536449    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2 podName:71e888ac-8fe5-4706-8af5-0c3227647e96 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:23.53643538 +0000 UTC m=+6.094780173 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vh7v2" (UniqueName: "kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2") pod "network-check-target-82qzb" (UID: "71e888ac-8fe5-4706-8af5-0c3227647e96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:21.852266 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:21.852152    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 22:41:19 +0000 UTC" deadline="2028-02-02 22:27:34.938174835 +0000 UTC"
Jul 29 22:46:21.852266 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:21.852188    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13271h41m13.085991544s"
Jul 29 22:46:21.947414 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:21.946909    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:21.947414 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:21.947036    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7pv5p" podUID="8687fc04-c1c6-4547-9787-dc94bfc700c3"
Jul 29 22:46:21.948040 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:21.947790    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:21.948040 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:21.947903    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82qzb" podUID="71e888ac-8fe5-4706-8af5-0c3227647e96"
Jul 29 22:46:21.996224 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:21.996192    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal" event={"ID":"6ceace035349ec6ca973ac8479406fd4","Type":"ContainerStarted","Data":"6b686609836b645bab825b81d75348191e01372cb50b30f55f84c95c3789b7eb"}
Jul 29 22:46:22.018706 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:22.018649    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-242.ec2.internal" podStartSLOduration=3.018612014 podStartE2EDuration="3.018612014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:22.011659869 +0000 UTC m=+4.570004686" watchObservedRunningTime="2026-07-29 22:46:22.018612014 +0000 UTC m=+4.576956830"
Jul 29 22:46:22.946500 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:22.946467    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:22.946967 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:22.946603    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:46:23.453452 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:23.452846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret\") pod \"global-pull-secret-syncer-7pv5p\" (UID: \"8687fc04-c1c6-4547-9787-dc94bfc700c3\") " pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:23.453452 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:23.453025    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:23.453452 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:23.453085    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret podName:8687fc04-c1c6-4547-9787-dc94bfc700c3 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:27.453068273 +0000 UTC m=+10.011413080 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret") pod "global-pull-secret-syncer-7pv5p" (UID: "8687fc04-c1c6-4547-9787-dc94bfc700c3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:23.553591 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:23.553216    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs\") pod \"network-metrics-daemon-2ksj4\" (UID: \"cf450310-9f32-4f65-96ea-b4d676ff2df7\") " pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:23.553591 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:23.553303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7v2\" (UniqueName: \"kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2\") pod \"network-check-target-82qzb\" (UID: \"71e888ac-8fe5-4706-8af5-0c3227647e96\") " pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:23.553591 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:23.553437    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:23.553591 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:23.553455    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:23.553591 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:23.553467    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vh7v2 for pod openshift-network-diagnostics/network-check-target-82qzb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:23.553591 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:23.553510    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:23.553591 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:23.553526    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2 podName:71e888ac-8fe5-4706-8af5-0c3227647e96 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:27.553508636 +0000 UTC m=+10.111853449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vh7v2" (UniqueName: "kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2") pod "network-check-target-82qzb" (UID: "71e888ac-8fe5-4706-8af5-0c3227647e96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:23.553591 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:23.553551    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs podName:cf450310-9f32-4f65-96ea-b4d676ff2df7 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:27.553539429 +0000 UTC m=+10.111884236 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs") pod "network-metrics-daemon-2ksj4" (UID: "cf450310-9f32-4f65-96ea-b4d676ff2df7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:23.947120 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:23.946880    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:23.947120 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:23.947016    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7pv5p" podUID="8687fc04-c1c6-4547-9787-dc94bfc700c3"
Jul 29 22:46:23.947611 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:23.947540    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:23.947674 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:23.947630    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82qzb" podUID="71e888ac-8fe5-4706-8af5-0c3227647e96"
Jul 29 22:46:24.946834 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:24.946306    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:24.946834 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:24.946452    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:46:25.947129 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:25.947104    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:25.947517 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:25.947219    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7pv5p" podUID="8687fc04-c1c6-4547-9787-dc94bfc700c3"
Jul 29 22:46:25.947517 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:25.947104    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:25.947517 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:25.947341    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82qzb" podUID="71e888ac-8fe5-4706-8af5-0c3227647e96"
Jul 29 22:46:26.947025 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:26.946990    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:26.947194 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:26.947129    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:46:27.485837 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:27.485786    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret\") pod \"global-pull-secret-syncer-7pv5p\" (UID: \"8687fc04-c1c6-4547-9787-dc94bfc700c3\") " pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:27.486017 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:27.485929    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:27.486017 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:27.485994    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret podName:8687fc04-c1c6-4547-9787-dc94bfc700c3 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:35.485974692 +0000 UTC m=+18.044319486 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret") pod "global-pull-secret-syncer-7pv5p" (UID: "8687fc04-c1c6-4547-9787-dc94bfc700c3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:27.586395 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:27.586354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs\") pod \"network-metrics-daemon-2ksj4\" (UID: \"cf450310-9f32-4f65-96ea-b4d676ff2df7\") " pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:27.586559 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:27.586458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7v2\" (UniqueName: \"kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2\") pod \"network-check-target-82qzb\" (UID: \"71e888ac-8fe5-4706-8af5-0c3227647e96\") " pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:27.586625 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:27.586594    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:27.586625 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:27.586614    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:27.586727 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:27.586627    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vh7v2 for pod openshift-network-diagnostics/network-check-target-82qzb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:27.586727 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:27.586681    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2 podName:71e888ac-8fe5-4706-8af5-0c3227647e96 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:35.586662295 +0000 UTC m=+18.145007095 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vh7v2" (UniqueName: "kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2") pod "network-check-target-82qzb" (UID: "71e888ac-8fe5-4706-8af5-0c3227647e96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:27.587123 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:27.587033    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:27.587123 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:27.587100    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs podName:cf450310-9f32-4f65-96ea-b4d676ff2df7 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:35.587081797 +0000 UTC m=+18.145426640 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs") pod "network-metrics-daemon-2ksj4" (UID: "cf450310-9f32-4f65-96ea-b4d676ff2df7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:27.948669 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:27.948041    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:27.948669 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:27.948150    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82qzb" podUID="71e888ac-8fe5-4706-8af5-0c3227647e96"
Jul 29 22:46:27.948669 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:27.948498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:27.948669 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:27.948581    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7pv5p" podUID="8687fc04-c1c6-4547-9787-dc94bfc700c3"
Jul 29 22:46:28.947566 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:28.947083    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:28.947566 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:28.947221    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:46:29.946914 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:29.946880    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:29.947317 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:29.946878    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:29.947317 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:29.947013    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7pv5p" podUID="8687fc04-c1c6-4547-9787-dc94bfc700c3"
Jul 29 22:46:29.947317 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:29.947084    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82qzb" podUID="71e888ac-8fe5-4706-8af5-0c3227647e96"
Jul 29 22:46:30.946328 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:30.946300    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:30.946466 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:30.946403    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:46:31.946504 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:31.946475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:31.946979 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:31.946514    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:31.946979 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:31.946590    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82qzb" podUID="71e888ac-8fe5-4706-8af5-0c3227647e96"
Jul 29 22:46:31.946979 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:31.946711    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7pv5p" podUID="8687fc04-c1c6-4547-9787-dc94bfc700c3"
Jul 29 22:46:32.946813 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:32.946772    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:32.947260 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:32.946884    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:46:33.946297 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:33.946260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:33.946466 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:33.946385    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82qzb" podUID="71e888ac-8fe5-4706-8af5-0c3227647e96"
Jul 29 22:46:33.946466 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:33.946447    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:33.946638 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:33.946548    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7pv5p" podUID="8687fc04-c1c6-4547-9787-dc94bfc700c3"
Jul 29 22:46:34.946153 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:34.946122    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:34.946551 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:34.946250    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:46:35.553355 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:35.553318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret\") pod \"global-pull-secret-syncer-7pv5p\" (UID: \"8687fc04-c1c6-4547-9787-dc94bfc700c3\") " pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:35.553540 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:35.553444    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:35.553540 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:35.553509    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret podName:8687fc04-c1c6-4547-9787-dc94bfc700c3 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:51.553492136 +0000 UTC m=+34.111836936 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret") pod "global-pull-secret-syncer-7pv5p" (UID: "8687fc04-c1c6-4547-9787-dc94bfc700c3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:35.654439 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:35.654407    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs\") pod \"network-metrics-daemon-2ksj4\" (UID: \"cf450310-9f32-4f65-96ea-b4d676ff2df7\") " pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:35.654614 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:35.654470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7v2\" (UniqueName: \"kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2\") pod \"network-check-target-82qzb\" (UID: \"71e888ac-8fe5-4706-8af5-0c3227647e96\") " pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:35.654614 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:35.654567    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:35.654614 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:35.654593    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:35.654614 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:35.654606    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vh7v2 for pod openshift-network-diagnostics/network-check-target-82qzb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:35.654834 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:35.654563    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:35.654834 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:35.654652    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2 podName:71e888ac-8fe5-4706-8af5-0c3227647e96 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:51.654639999 +0000 UTC m=+34.212984792 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vh7v2" (UniqueName: "kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2") pod "network-check-target-82qzb" (UID: "71e888ac-8fe5-4706-8af5-0c3227647e96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:35.654834 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:35.654709    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs podName:cf450310-9f32-4f65-96ea-b4d676ff2df7 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:51.654692307 +0000 UTC m=+34.213037116 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs") pod "network-metrics-daemon-2ksj4" (UID: "cf450310-9f32-4f65-96ea-b4d676ff2df7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:35.946663 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:35.946586    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:35.946663 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:35.946630    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:35.947172 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:35.946708    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82qzb" podUID="71e888ac-8fe5-4706-8af5-0c3227647e96"
Jul 29 22:46:35.947172 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:35.946819    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7pv5p" podUID="8687fc04-c1c6-4547-9787-dc94bfc700c3"
Jul 29 22:46:36.946542 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:36.946511    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:36.946723 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:36.946651    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:46:37.946760 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:37.946730    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:37.947246 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:37.946839    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82qzb" podUID="71e888ac-8fe5-4706-8af5-0c3227647e96"
Jul 29 22:46:37.947246 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:37.946914    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:37.947246 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:37.947010    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7pv5p" podUID="8687fc04-c1c6-4547-9787-dc94bfc700c3"
Jul 29 22:46:38.946875 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:38.946847    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:38.947217 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:38.946941    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:46:39.951206 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:39.950724    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:39.952133 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:39.950777    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:39.952133 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:39.951287    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7pv5p" podUID="8687fc04-c1c6-4547-9787-dc94bfc700c3"
Jul 29 22:46:39.952133 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:39.951324    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82qzb" podUID="71e888ac-8fe5-4706-8af5-0c3227647e96"
Jul 29 22:46:40.036650 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.036618    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6dn6d" event={"ID":"9a191d3f-9334-4881-8380-654a3b33d7d0","Type":"ContainerStarted","Data":"34f63ff0b97c64a20427302546cf5bcdfebd388583f9077ab4efd20359e341bc"}
Jul 29 22:46:40.038213 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.038189    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-whv2s" event={"ID":"0ad705ff-e172-497a-a972-108f30e7be6b","Type":"ContainerStarted","Data":"d346062e012f47407bfd0c1d0a40b05ee27d869920179ee4fc9b51cc0251000b"}
Jul 29 22:46:40.040939 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.040914    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl" event={"ID":"6166a6a8-c059-4854-a4c7-51904b3265d6","Type":"ContainerStarted","Data":"3acfd71a9963f3c120d8f63407dd05178df249efb0850f558e069762ed8aaed0"}
Jul 29 22:46:40.041038 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.040974    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl" event={"ID":"6166a6a8-c059-4854-a4c7-51904b3265d6","Type":"ContainerStarted","Data":"f426d74105d14b2ef661d116e6ac9fcfee8c531f3d186808cc9f00515bf6f625"}
Jul 29 22:46:40.041038 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.041011    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl" event={"ID":"6166a6a8-c059-4854-a4c7-51904b3265d6","Type":"ContainerStarted","Data":"96a22677b91ac05ea99b4b396422d64752f2a593c51643e3fdc30d958ce2a88a"}
Jul 29 22:46:40.041119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.041052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl" event={"ID":"6166a6a8-c059-4854-a4c7-51904b3265d6","Type":"ContainerStarted","Data":"505dcee5ab4804762d623336ea10783d23bac034626fd74976dd7dd8c2c459b7"}
Jul 29 22:46:40.041119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.041091    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl" event={"ID":"6166a6a8-c059-4854-a4c7-51904b3265d6","Type":"ContainerStarted","Data":"c14657e5b260017e3e039ff39e1248efdd6fc873ebe419c3421a980e5d875036"}
Jul 29 22:46:40.041190 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.041136    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl" event={"ID":"6166a6a8-c059-4854-a4c7-51904b3265d6","Type":"ContainerStarted","Data":"ce5d7a0e6ff4747101a8fc0312eb4e4df01e60c3b467ec1f0afc58f612475a48"}
Jul 29 22:46:40.042242 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.042223    2581 generic.go:356] "Generic (PLEG): container finished" podID="0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8" containerID="c1e278bac497ee2484a51dddbf63fa29503e4f56855beb9f2b13116dc8fe109c" exitCode=0
Jul 29 22:46:40.042324 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.042292    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z5plq" event={"ID":"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8","Type":"ContainerDied","Data":"c1e278bac497ee2484a51dddbf63fa29503e4f56855beb9f2b13116dc8fe109c"}
Jul 29 22:46:40.043699 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.043680    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-65x8z" event={"ID":"950bb527-68a7-42b6-8aa4-b5242645d169","Type":"ContainerStarted","Data":"4d2b4686e8e634210319e3d10ba8c9d3aaac8f41850455fd893265385932a710"}
Jul 29 22:46:40.045080 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.045060    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9cmk4" event={"ID":"19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96","Type":"ContainerStarted","Data":"e8d95ee53872844a640f920ee0ccb6b965221082dcaca814e68e02fe646e7a1f"}
Jul 29 22:46:40.046217 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.046200    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq" event={"ID":"85bca852-e00d-4a1a-a746-11bd4cd94508","Type":"ContainerStarted","Data":"34c71b3583d0141a922faa15a33a8257f2b36b3c2a9a29a07cf700f55b790977"}
Jul 29 22:46:40.047375 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.047355    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8tftg" event={"ID":"bb349f88-27d7-4900-9d7c-d0e7916098ab","Type":"ContainerStarted","Data":"8005fcd78f73d236ec0234d6c89aaf231c0c0c6020f3cb1aa023b116e58c494d"}
Jul 29 22:46:40.070218 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.070173    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9cmk4" podStartSLOduration=4.554686224 podStartE2EDuration="23.070155411s" totalImagesPullingTime="18.515469187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:40.069352195 +0000 UTC m=+22.627697011" watchObservedRunningTime="2026-07-29 22:46:40.070155411 +0000 UTC m=+22.628500228"
Jul 29 22:46:40.070388 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.070362    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6dn6d" podStartSLOduration=3.5889715620000002 podStartE2EDuration="22.070355044s" totalImagesPullingTime="18.481383482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:40.051557022 +0000 UTC m=+22.609901838" watchObservedRunningTime="2026-07-29 22:46:40.070355044 +0000 UTC m=+22.628699862"
Jul 29 22:46:40.085121 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.085085    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-65x8z" podStartSLOduration=3.613160749 podStartE2EDuration="22.08507383s" totalImagesPullingTime="18.471913081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:40.085057234 +0000 UTC m=+22.643402049" watchObservedRunningTime="2026-07-29 22:46:40.08507383 +0000 UTC m=+22.643418646"
Jul 29 22:46:40.132690 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.132656    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8tftg" podStartSLOduration=3.651700047 podStartE2EDuration="22.132644623s" totalImagesPullingTime="18.480944576s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:40.132281019 +0000 UTC m=+22.690625837" watchObservedRunningTime="2026-07-29 22:46:40.132644623 +0000 UTC m=+22.690989439"
Jul 29 22:46:40.693675 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.693646    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 22:46:40.882356 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.882163    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T22:46:40.69366654Z","UUID":"fe313896-48fe-487f-963d-23850fd578f1","Handler":null,"Name":"","Endpoint":""}
Jul 29 22:46:40.883847 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.883827    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 22:46:40.883979 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.883867    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 22:46:40.946138 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:40.946108    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:40.946322 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:40.946265    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:46:41.051671 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:41.051644    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq" event={"ID":"85bca852-e00d-4a1a-a746-11bd4cd94508","Type":"ContainerStarted","Data":"c9b10add4176c81cd758405394621bc653fcb4506b7a9a71275b96c693b9ced0"}
Jul 29 22:46:41.053279 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:41.053254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c4d6f" event={"ID":"556818de-e974-4303-80b3-846815ab4cd3","Type":"ContainerStarted","Data":"8bf15920ab5cdeb37d814c00dce61f8e18df75ef333f71ac655630306e597305"}
Jul 29 22:46:41.070114 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:41.070050    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-whv2s" podStartSLOduration=4.580843501 podStartE2EDuration="23.070033293s" totalImagesPullingTime="18.489189792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:40.148279824 +0000 UTC m=+22.706624640" watchObservedRunningTime="2026-07-29 22:46:41.070033293 +0000 UTC m=+23.628378110"
Jul 29 22:46:41.070293 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:41.070273    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c4d6f" podStartSLOduration=4.594388041 podStartE2EDuration="23.070267927s" totalImagesPullingTime="18.475879886s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:41.069846255 +0000 UTC m=+23.628191070" watchObservedRunningTime="2026-07-29 22:46:41.070267927 +0000 UTC m=+23.628612743"
Jul 29 22:46:41.946859 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:41.946838    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:41.947010 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:41.946952    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82qzb" podUID="71e888ac-8fe5-4706-8af5-0c3227647e96"
Jul 29 22:46:41.947081 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:41.947061    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:41.947202 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:41.947146    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7pv5p" podUID="8687fc04-c1c6-4547-9787-dc94bfc700c3"
Jul 29 22:46:42.061612 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:42.061579    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl" event={"ID":"6166a6a8-c059-4854-a4c7-51904b3265d6","Type":"ContainerStarted","Data":"2c0b9c417c56b7af452641ef385e0cf0eb80b31041c8f5463b4abae4c7b5fd17"}
Jul 29 22:46:42.064319 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:42.064294    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq" event={"ID":"85bca852-e00d-4a1a-a746-11bd4cd94508","Type":"ContainerStarted","Data":"7b6045a4d2de0adceab4208b042eb47d01ad52e6a6da8d5b2fb73f48b941e317"}
Jul 29 22:46:42.082984 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:42.082893    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhqrq" podStartSLOduration=4.272133331 podStartE2EDuration="25.082874365s" totalImagesPullingTime="20.810741034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:42.082089265 +0000 UTC m=+24.640434082" watchObservedRunningTime="2026-07-29 22:46:42.082874365 +0000 UTC m=+24.641219184"
Jul 29 22:46:42.946394 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:42.946362    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:42.946550 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:42.946501    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:46:43.500454 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:43.500220    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6dn6d"
Jul 29 22:46:43.500869 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:43.500786    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6dn6d"
Jul 29 22:46:43.946979 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:43.946908    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:43.947137 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:43.946908    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:43.947137 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:43.947028    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7pv5p" podUID="8687fc04-c1c6-4547-9787-dc94bfc700c3"
Jul 29 22:46:43.947137 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:43.947114    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82qzb" podUID="71e888ac-8fe5-4706-8af5-0c3227647e96"
Jul 29 22:46:44.068838 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:44.068791    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6dn6d"
Jul 29 22:46:44.069384 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:44.069364    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6dn6d"
Jul 29 22:46:44.946919 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:44.946891    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:44.947682 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:44.947008    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:46:45.073906 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:45.073878    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl" event={"ID":"6166a6a8-c059-4854-a4c7-51904b3265d6","Type":"ContainerStarted","Data":"05c9ece5bca5b731894680d7c8a405c466761853a908b3cbf7b6a38329c8e453"}
Jul 29 22:46:45.074101 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:45.074080    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:45.074182 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:45.074158    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:45.075735 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:45.075712    2581 generic.go:356] "Generic (PLEG): container finished" podID="0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8" containerID="79b5e561d4736ccb79619eb771486faf8c40d1df43fd270d05868e8f41fa630e" exitCode=0
Jul 29 22:46:45.075840 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:45.075780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z5plq" event={"ID":"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8","Type":"ContainerDied","Data":"79b5e561d4736ccb79619eb771486faf8c40d1df43fd270d05868e8f41fa630e"}
Jul 29 22:46:45.089488 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:45.089472    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:45.106043 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:45.106008    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl" podStartSLOduration=8.566441465 podStartE2EDuration="27.105996502s" totalImagesPullingTime="18.539555037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:45.10568114 +0000 UTC m=+27.664025955" watchObservedRunningTime="2026-07-29 22:46:45.105996502 +0000 UTC m=+27.664341318"
Jul 29 22:46:45.946383 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:45.946217    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:45.946502 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:45.946273    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:45.946502 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:45.946455    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82qzb" podUID="71e888ac-8fe5-4706-8af5-0c3227647e96"
Jul 29 22:46:45.946574 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:45.946542    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7pv5p" podUID="8687fc04-c1c6-4547-9787-dc94bfc700c3"
Jul 29 22:46:46.077689 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:46.077657    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:46.095392 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:46.095376    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:46:46.151344 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:46.151320    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7pv5p"]
Jul 29 22:46:46.151453 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:46.151434    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:46.151576 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:46.151539    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7pv5p" podUID="8687fc04-c1c6-4547-9787-dc94bfc700c3"
Jul 29 22:46:46.154688 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:46.154665    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2ksj4"]
Jul 29 22:46:46.154812 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:46.154771    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:46.155641 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:46.154888    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:46:46.155641 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:46.155432    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-82qzb"]
Jul 29 22:46:46.155641 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:46.155512    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:46.155641 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:46.155594    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82qzb" podUID="71e888ac-8fe5-4706-8af5-0c3227647e96"
Jul 29 22:46:47.081593 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:47.081562    2581 generic.go:356] "Generic (PLEG): container finished" podID="0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8" containerID="7972d0bcdb68ba2a9dfa30ee59a8e4daa26a3dc1dc82f772fe17e08e2bf53071" exitCode=0
Jul 29 22:46:47.082069 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:47.081641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z5plq" event={"ID":"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8","Type":"ContainerDied","Data":"7972d0bcdb68ba2a9dfa30ee59a8e4daa26a3dc1dc82f772fe17e08e2bf53071"}
Jul 29 22:46:47.950191 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:47.950164    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:47.950370 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:47.950208    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:47.950370 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:47.950288    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:47.950370 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:47.950304    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:46:47.950514 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:47.950375    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82qzb" podUID="71e888ac-8fe5-4706-8af5-0c3227647e96"
Jul 29 22:46:47.950514 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:47.950451    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7pv5p" podUID="8687fc04-c1c6-4547-9787-dc94bfc700c3"
Jul 29 22:46:49.088338 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:49.088308    2581 generic.go:356] "Generic (PLEG): container finished" podID="0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8" containerID="f01ec29ac277716f20b4a77b809ab65ee0ce7369b5c97fc6ea526234f332d264" exitCode=0
Jul 29 22:46:49.088656 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:49.088355    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z5plq" event={"ID":"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8","Type":"ContainerDied","Data":"f01ec29ac277716f20b4a77b809ab65ee0ce7369b5c97fc6ea526234f332d264"}
Jul 29 22:46:49.946337 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:49.946274    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:49.946507 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:49.946279    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:49.946507 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:49.946409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:49.946507 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:49.946405    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7pv5p" podUID="8687fc04-c1c6-4547-9787-dc94bfc700c3"
Jul 29 22:46:49.946652 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:49.946512    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:46:49.946652 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:49.946588    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82qzb" podUID="71e888ac-8fe5-4706-8af5-0c3227647e96"
Jul 29 22:46:51.577516 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.574918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret\") pod \"global-pull-secret-syncer-7pv5p\" (UID: \"8687fc04-c1c6-4547-9787-dc94bfc700c3\") " pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:51.577516 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:51.575402    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:51.577516 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:51.575469    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret podName:8687fc04-c1c6-4547-9787-dc94bfc700c3 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:23.575450512 +0000 UTC m=+66.133795310 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret") pod "global-pull-secret-syncer-7pv5p" (UID: "8687fc04-c1c6-4547-9787-dc94bfc700c3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:51.676316 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.676288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7v2\" (UniqueName: \"kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2\") pod \"network-check-target-82qzb\" (UID: \"71e888ac-8fe5-4706-8af5-0c3227647e96\") " pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:51.676464 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.676329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs\") pod \"network-metrics-daemon-2ksj4\" (UID: \"cf450310-9f32-4f65-96ea-b4d676ff2df7\") " pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:51.676464 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:51.676433    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:51.676539 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:51.676483    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs podName:cf450310-9f32-4f65-96ea-b4d676ff2df7 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:23.676469064 +0000 UTC m=+66.234813861 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs") pod "network-metrics-daemon-2ksj4" (UID: "cf450310-9f32-4f65-96ea-b4d676ff2df7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:51.676539 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:51.676482    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:51.676539 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:51.676509    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:51.676539 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:51.676524    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vh7v2 for pod openshift-network-diagnostics/network-check-target-82qzb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:51.676679 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:51.676595    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2 podName:71e888ac-8fe5-4706-8af5-0c3227647e96 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:23.676577836 +0000 UTC m=+66.234922644 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vh7v2" (UniqueName: "kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2") pod "network-check-target-82qzb" (UID: "71e888ac-8fe5-4706-8af5-0c3227647e96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:51.805915 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.805892    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-242.ec2.internal" event="NodeReady"
Jul 29 22:46:51.806061 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.806031    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 22:46:51.840445 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.840196    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c676dc74-4k7lf"]
Jul 29 22:46:51.843233 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.843204    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"]
Jul 29 22:46:51.843365 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.843350    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:51.844936 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.844914    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:46:51.846221 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.846182    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 22:46:51.846309 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.846203    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 22:46:51.846309 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.846203    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 22:46:51.846309 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.846284    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vpnhm\""
Jul 29 22:46:51.847613 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.847591    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 29 22:46:51.848043 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.847874    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 29 22:46:51.849102 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.849082    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-trcnk\""
Jul 29 22:46:51.853603 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.853576    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 22:46:51.854740 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.854719    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"]
Jul 29 22:46:51.857099 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.857078    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-25gll"]
Jul 29 22:46:51.859985 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.859868    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c676dc74-4k7lf"]
Jul 29 22:46:51.860085 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.860001    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-25gll"
Jul 29 22:46:51.861101 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.861079    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hlgwc"]
Jul 29 22:46:51.862751 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.862720    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hlgwc"
Jul 29 22:46:51.863100 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.862981    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 22:46:51.863100 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.863048    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 22:46:51.863331 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.863312    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jtttq\""
Jul 29 22:46:51.865717 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.865691    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 22:46:51.865826 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.865691    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d8h7n\""
Jul 29 22:46:51.865976 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.865951    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 22:46:51.866051 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.865951    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 22:46:51.870129 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.870106    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-25gll"]
Jul 29 22:46:51.873840 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.873823    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hlgwc"]
Jul 29 22:46:51.946951 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.946917    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:46:51.947117 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.946919    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:46:51.947439 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.946923    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:46:51.950156 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.950138    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 22:46:51.950262 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.950159    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 22:46:51.950262 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.950179    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wkbjs\""
Jul 29 22:46:51.950374 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.950285    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 22:46:51.950374 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.950302    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-csbtv\""
Jul 29 22:46:51.950545 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.950529    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 22:46:51.980825 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.979090    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7e081dc-a705-4255-aed8-c20f7090c5b3-ca-trust-extracted\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:51.980825 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.979207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:51.980825 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.979238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7e081dc-a705-4255-aed8-c20f7090c5b3-trusted-ca\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:51.980825 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.979274    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab3523d2-8acf-462d-8458-3b2e6fa74f08-tmp-dir\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:46:51.980825 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.979340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7e081dc-a705-4255-aed8-c20f7090c5b3-installation-pull-secrets\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:51.980825 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.979416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3523d2-8acf-462d-8458-3b2e6fa74f08-config-volume\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:46:51.980825 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.979459    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxgsb\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-kube-api-access-zxgsb\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:51.980825 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.979505    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwkrv\" (UniqueName: \"kubernetes.io/projected/81d4f0dc-2d3a-4523-83d4-de99adcf5e03-kube-api-access-vwkrv\") pod \"ingress-canary-hlgwc\" (UID: \"81d4f0dc-2d3a-4523-83d4-de99adcf5e03\") " pod="openshift-ingress-canary/ingress-canary-hlgwc"
Jul 29 22:46:51.980825 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.979538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-certificates\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:51.980825 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.979600    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:46:51.980825 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.979625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81d4f0dc-2d3a-4523-83d4-de99adcf5e03-cert\") pod \"ingress-canary-hlgwc\" (UID: \"81d4f0dc-2d3a-4523-83d4-de99adcf5e03\") " pod="openshift-ingress-canary/ingress-canary-hlgwc"
Jul 29 22:46:51.980825 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.979709    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-sk5qv\" (UID: \"2fe2a7d4-0e16-4f28-9173-d5765b53e33b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:46:51.980825 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.979744    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7e081dc-a705-4255-aed8-c20f7090c5b3-image-registry-private-configuration\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:51.980825 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.979819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-bound-sa-token\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:51.980825 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.979854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrkpk\" (UniqueName: \"kubernetes.io/projected/ab3523d2-8acf-462d-8458-3b2e6fa74f08-kube-api-access-nrkpk\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:46:51.981507 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:51.979941    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sk5qv\" (UID: \"2fe2a7d4-0e16-4f28-9173-d5765b53e33b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:46:52.012841 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.012793    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:46:52.080794 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.080767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-sk5qv\" (UID: \"2fe2a7d4-0e16-4f28-9173-d5765b53e33b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:46:52.080923 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.080816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7e081dc-a705-4255-aed8-c20f7090c5b3-image-registry-private-configuration\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:52.080923 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.080873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-bound-sa-token\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:52.080923 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.080912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkpk\" (UniqueName: \"kubernetes.io/projected/ab3523d2-8acf-462d-8458-3b2e6fa74f08-kube-api-access-nrkpk\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:46:52.081064 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.080938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sk5qv\" (UID: \"2fe2a7d4-0e16-4f28-9173-d5765b53e33b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:46:52.081064 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.080983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7e081dc-a705-4255-aed8-c20f7090c5b3-ca-trust-extracted\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:52.081064 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.081010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:52.081064 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.081031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7e081dc-a705-4255-aed8-c20f7090c5b3-trusted-ca\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:52.081064 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.081054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab3523d2-8acf-462d-8458-3b2e6fa74f08-tmp-dir\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:46:52.081280 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:52.081103    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 22:46:52.081280 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.081120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7e081dc-a705-4255-aed8-c20f7090c5b3-installation-pull-secrets\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:52.081280 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.081145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3523d2-8acf-462d-8458-3b2e6fa74f08-config-volume\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:46:52.081280 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:52.081173    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert podName:2fe2a7d4-0e16-4f28-9173-d5765b53e33b nodeName:}" failed. No retries permitted until 2026-07-29 22:46:52.581153097 +0000 UTC m=+35.139497894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sk5qv" (UID: "2fe2a7d4-0e16-4f28-9173-d5765b53e33b") : secret "networking-console-plugin-cert" not found
Jul 29 22:46:52.081280 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.081226    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxgsb\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-kube-api-access-zxgsb\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:52.081280 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.081256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwkrv\" (UniqueName: \"kubernetes.io/projected/81d4f0dc-2d3a-4523-83d4-de99adcf5e03-kube-api-access-vwkrv\") pod \"ingress-canary-hlgwc\" (UID: \"81d4f0dc-2d3a-4523-83d4-de99adcf5e03\") " pod="openshift-ingress-canary/ingress-canary-hlgwc"
Jul 29 22:46:52.081533 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.081287    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-certificates\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:52.081533 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.081324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:46:52.081533 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.081344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81d4f0dc-2d3a-4523-83d4-de99adcf5e03-cert\") pod \"ingress-canary-hlgwc\" (UID: \"81d4f0dc-2d3a-4523-83d4-de99adcf5e03\") " pod="openshift-ingress-canary/ingress-canary-hlgwc"
Jul 29 22:46:52.081533 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.081354    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7e081dc-a705-4255-aed8-c20f7090c5b3-ca-trust-extracted\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:52.081533 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.081423    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab3523d2-8acf-462d-8458-3b2e6fa74f08-tmp-dir\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:46:52.081533 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:52.081508    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 22:46:52.081533 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:52.081521    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c676dc74-4k7lf: secret "image-registry-tls" not found
Jul 29 22:46:52.081907 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:52.081564    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls podName:d7e081dc-a705-4255-aed8-c20f7090c5b3 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:52.58154791 +0000 UTC m=+35.139892713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls") pod "image-registry-5c676dc74-4k7lf" (UID: "d7e081dc-a705-4255-aed8-c20f7090c5b3") : secret "image-registry-tls" not found
Jul 29 22:46:52.081907 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:52.081719    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:46:52.081907 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:52.081791    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls podName:ab3523d2-8acf-462d-8458-3b2e6fa74f08 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:52.581768789 +0000 UTC m=+35.140113597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls") pod "dns-default-25gll" (UID: "ab3523d2-8acf-462d-8458-3b2e6fa74f08") : secret "dns-default-metrics-tls" not found
Jul 29 22:46:52.082138 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.082110    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-certificates\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:52.082408 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.082388    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-sk5qv\" (UID: \"2fe2a7d4-0e16-4f28-9173-d5765b53e33b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:46:52.082493 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.082433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7e081dc-a705-4255-aed8-c20f7090c5b3-trusted-ca\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:52.082541 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.082487    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3523d2-8acf-462d-8458-3b2e6fa74f08-config-volume\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:46:52.085053 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.085034    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7e081dc-a705-4255-aed8-c20f7090c5b3-installation-pull-secrets\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:52.085194 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.085171    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81d4f0dc-2d3a-4523-83d4-de99adcf5e03-cert\") pod \"ingress-canary-hlgwc\" (UID: \"81d4f0dc-2d3a-4523-83d4-de99adcf5e03\") " pod="openshift-ingress-canary/ingress-canary-hlgwc"
Jul 29 22:46:52.089089 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.089069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7e081dc-a705-4255-aed8-c20f7090c5b3-image-registry-private-configuration\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:52.089673 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.089653    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-bound-sa-token\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:52.097157 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.097110    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwkrv\" (UniqueName: \"kubernetes.io/projected/81d4f0dc-2d3a-4523-83d4-de99adcf5e03-kube-api-access-vwkrv\") pod \"ingress-canary-hlgwc\" (UID: \"81d4f0dc-2d3a-4523-83d4-de99adcf5e03\") " pod="openshift-ingress-canary/ingress-canary-hlgwc"
Jul 29 22:46:52.097333 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.097274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkpk\" (UniqueName: \"kubernetes.io/projected/ab3523d2-8acf-462d-8458-3b2e6fa74f08-kube-api-access-nrkpk\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:46:52.097718 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.097697    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxgsb\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-kube-api-access-zxgsb\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:52.113530 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.113487    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:46:52.178740 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.178709    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hlgwc"
Jul 29 22:46:52.585409 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.585376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:46:52.586049 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.585435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sk5qv\" (UID: \"2fe2a7d4-0e16-4f28-9173-d5765b53e33b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:46:52.586049 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:52.585470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:52.586049 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:52.585544    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:46:52.586049 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:52.585566    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 22:46:52.586049 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:52.585578    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c676dc74-4k7lf: secret "image-registry-tls" not found
Jul 29 22:46:52.586049 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:52.585600    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 22:46:52.586049 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:52.585622    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls podName:ab3523d2-8acf-462d-8458-3b2e6fa74f08 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:53.585600655 +0000 UTC m=+36.143945454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls") pod "dns-default-25gll" (UID: "ab3523d2-8acf-462d-8458-3b2e6fa74f08") : secret "dns-default-metrics-tls" not found
Jul 29 22:46:52.586049 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:52.585641    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls podName:d7e081dc-a705-4255-aed8-c20f7090c5b3 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:53.585631604 +0000 UTC m=+36.143976405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls") pod "image-registry-5c676dc74-4k7lf" (UID: "d7e081dc-a705-4255-aed8-c20f7090c5b3") : secret "image-registry-tls" not found
Jul 29 22:46:52.586049 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:52.585655    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert podName:2fe2a7d4-0e16-4f28-9173-d5765b53e33b nodeName:}" failed. No retries permitted until 2026-07-29 22:46:53.585648336 +0000 UTC m=+36.143993132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sk5qv" (UID: "2fe2a7d4-0e16-4f28-9173-d5765b53e33b") : secret "networking-console-plugin-cert" not found
Jul 29 22:46:53.318500 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:53.318368    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hlgwc"]
Jul 29 22:46:53.321745 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:46:53.321717    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d4f0dc_2d3a_4523_83d4_de99adcf5e03.slice/crio-4857a62851b8bccc6139531556a51830fdfde08a01853859e51eb9b3700b5a7e WatchSource:0}: Error finding container 4857a62851b8bccc6139531556a51830fdfde08a01853859e51eb9b3700b5a7e: Status 404 returned error can't find the container with id 4857a62851b8bccc6139531556a51830fdfde08a01853859e51eb9b3700b5a7e
Jul 29 22:46:53.594601 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:53.594575    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:46:53.595125 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:53.594614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sk5qv\" (UID: \"2fe2a7d4-0e16-4f28-9173-d5765b53e33b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:46:53.595125 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:53.594644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:53.595125 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:53.594730    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 22:46:53.595125 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:53.594793    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert podName:2fe2a7d4-0e16-4f28-9173-d5765b53e33b nodeName:}" failed. No retries permitted until 2026-07-29 22:46:55.594777038 +0000 UTC m=+38.153121833 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sk5qv" (UID: "2fe2a7d4-0e16-4f28-9173-d5765b53e33b") : secret "networking-console-plugin-cert" not found
Jul 29 22:46:53.595125 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:53.594734    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 22:46:53.595125 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:53.594833    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c676dc74-4k7lf: secret "image-registry-tls" not found
Jul 29 22:46:53.595125 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:53.594872    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls podName:d7e081dc-a705-4255-aed8-c20f7090c5b3 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:55.59486062 +0000 UTC m=+38.153205446 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls") pod "image-registry-5c676dc74-4k7lf" (UID: "d7e081dc-a705-4255-aed8-c20f7090c5b3") : secret "image-registry-tls" not found
Jul 29 22:46:53.595125 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:53.594730    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:46:53.595125 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:53.594900    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls podName:ab3523d2-8acf-462d-8458-3b2e6fa74f08 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:55.594893836 +0000 UTC m=+38.153238629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls") pod "dns-default-25gll" (UID: "ab3523d2-8acf-462d-8458-3b2e6fa74f08") : secret "dns-default-metrics-tls" not found
Jul 29 22:46:54.103102 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:54.103074    2581 generic.go:356] "Generic (PLEG): container finished" podID="0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8" containerID="40b292167d510cf6f41d826f7e9803ba2b4dde7209b081516dd1a556ee29599b" exitCode=0
Jul 29 22:46:54.103247 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:54.103143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z5plq" event={"ID":"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8","Type":"ContainerDied","Data":"40b292167d510cf6f41d826f7e9803ba2b4dde7209b081516dd1a556ee29599b"}
Jul 29 22:46:54.104178 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:54.104159    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hlgwc" event={"ID":"81d4f0dc-2d3a-4523-83d4-de99adcf5e03","Type":"ContainerStarted","Data":"4857a62851b8bccc6139531556a51830fdfde08a01853859e51eb9b3700b5a7e"}
Jul 29 22:46:55.110735 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:55.110690    2581 generic.go:356] "Generic (PLEG): container finished" podID="0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8" containerID="82aec51a242e783e46d6f317216e3a9d5fda2686e7842b9a5c274a2f7bd601aa" exitCode=0
Jul 29 22:46:55.111179 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:55.110754    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z5plq" event={"ID":"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8","Type":"ContainerDied","Data":"82aec51a242e783e46d6f317216e3a9d5fda2686e7842b9a5c274a2f7bd601aa"}
Jul 29 22:46:55.609777 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:55.609739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:46:55.610032 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:55.609790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sk5qv\" (UID: \"2fe2a7d4-0e16-4f28-9173-d5765b53e33b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:46:55.610032 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:55.609847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:55.610032 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:55.609941    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:46:55.610032 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:55.609993    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 22:46:55.610032 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:55.610018    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls podName:ab3523d2-8acf-462d-8458-3b2e6fa74f08 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:59.609998533 +0000 UTC m=+42.168343341 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls") pod "dns-default-25gll" (UID: "ab3523d2-8acf-462d-8458-3b2e6fa74f08") : secret "dns-default-metrics-tls" not found
Jul 29 22:46:55.610299 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:55.610058    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert podName:2fe2a7d4-0e16-4f28-9173-d5765b53e33b nodeName:}" failed. No retries permitted until 2026-07-29 22:46:59.610040504 +0000 UTC m=+42.168385312 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sk5qv" (UID: "2fe2a7d4-0e16-4f28-9173-d5765b53e33b") : secret "networking-console-plugin-cert" not found
Jul 29 22:46:55.610299 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:55.609996    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 22:46:55.610299 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:55.610080    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c676dc74-4k7lf: secret "image-registry-tls" not found
Jul 29 22:46:55.610299 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:55.610112    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls podName:d7e081dc-a705-4255-aed8-c20f7090c5b3 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:59.61010128 +0000 UTC m=+42.168446077 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls") pod "image-registry-5c676dc74-4k7lf" (UID: "d7e081dc-a705-4255-aed8-c20f7090c5b3") : secret "image-registry-tls" not found
Jul 29 22:46:55.621938 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:55.621912    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:46:55.695780 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:55.695745    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:46:55.792574 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:55.792545    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:46:55.889731 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:55.889676    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:46:55.943009 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:55.942984    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:46:56.115941 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:56.115918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z5plq" event={"ID":"0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8","Type":"ContainerStarted","Data":"8ee359dc3bb9d1cfc553ac9ac0cf9d15408cd05a1ae06bb92cbdd10f14b0fc8f"}
Jul 29 22:46:56.141359 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:56.141214    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-z5plq" podStartSLOduration=8.927111276 podStartE2EDuration="38.141201792s" totalImagesPullingTime="29.214090516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:56.139501283 +0000 UTC m=+38.697846102" watchObservedRunningTime="2026-07-29 22:46:56.141201792 +0000 UTC m=+38.699546607"
Jul 29 22:46:56.264550 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:56.264398    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:46:56.321353 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:56.321317    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:46:56.495162 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:56.495092    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:46:56.758143 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:56.758068    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:46:57.119736 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:57.119705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hlgwc" event={"ID":"81d4f0dc-2d3a-4523-83d4-de99adcf5e03","Type":"ContainerStarted","Data":"59b562d2af55ae379333114522aa9aea3c50b46ae059545e8228dfa16d071ca6"}
Jul 29 22:46:57.137406 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:57.137358    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hlgwc" podStartSLOduration=3.362878694 podStartE2EDuration="6.137343232s" totalImagesPullingTime="2.774464538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:57.136773116 +0000 UTC m=+39.695117936" watchObservedRunningTime="2026-07-29 22:46:57.137343232 +0000 UTC m=+39.695688048"
Jul 29 22:46:57.157994 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:57.157968    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:46:57.220760 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:57.220736    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:46:58.565580 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:58.565544    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:46:58.628691 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:58.628658    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:46:59.636060 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:59.636026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sk5qv\" (UID: \"2fe2a7d4-0e16-4f28-9173-d5765b53e33b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:46:59.636424 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:59.636070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:46:59.636424 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:59.636160    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 22:46:59.636424 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:59.636188    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 22:46:59.636424 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:46:59.636189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:46:59.636424 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:59.636201    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c676dc74-4k7lf: secret "image-registry-tls" not found
Jul 29 22:46:59.636424 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:59.636215    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert podName:2fe2a7d4-0e16-4f28-9173-d5765b53e33b nodeName:}" failed. No retries permitted until 2026-07-29 22:47:07.636198888 +0000 UTC m=+50.194543682 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sk5qv" (UID: "2fe2a7d4-0e16-4f28-9173-d5765b53e33b") : secret "networking-console-plugin-cert" not found
Jul 29 22:46:59.636424 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:59.636249    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls podName:d7e081dc-a705-4255-aed8-c20f7090c5b3 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:07.636233117 +0000 UTC m=+50.194577914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls") pod "image-registry-5c676dc74-4k7lf" (UID: "d7e081dc-a705-4255-aed8-c20f7090c5b3") : secret "image-registry-tls" not found
Jul 29 22:46:59.636424 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:59.636260    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:46:59.636424 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:46:59.636298    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls podName:ab3523d2-8acf-462d-8458-3b2e6fa74f08 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:07.636287199 +0000 UTC m=+50.194632015 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls") pod "dns-default-25gll" (UID: "ab3523d2-8acf-462d-8458-3b2e6fa74f08") : secret "dns-default-metrics-tls" not found
Jul 29 22:47:00.847460 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:00.847428    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:47:02.910947 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:02.910912    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:47:02.991281 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:02.991251    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:47:04.186403 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:04.186362    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:47:04.239592 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:04.239545    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:47:07.697366 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:07.697330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sk5qv\" (UID: \"2fe2a7d4-0e16-4f28-9173-d5765b53e33b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:47:07.697366 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:07.697381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:47:07.697869 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:07.697431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:47:07.697869 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:07.697475    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 22:47:07.697869 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:07.697533    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert podName:2fe2a7d4-0e16-4f28-9173-d5765b53e33b nodeName:}" failed. No retries permitted until 2026-07-29 22:47:23.69751856 +0000 UTC m=+66.255863358 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sk5qv" (UID: "2fe2a7d4-0e16-4f28-9173-d5765b53e33b") : secret "networking-console-plugin-cert" not found
Jul 29 22:47:07.697869 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:07.697542    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 22:47:07.697869 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:07.697564    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c676dc74-4k7lf: secret "image-registry-tls" not found
Jul 29 22:47:07.697869 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:07.697580    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:47:07.697869 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:07.697612    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls podName:d7e081dc-a705-4255-aed8-c20f7090c5b3 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:23.697599948 +0000 UTC m=+66.255944746 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls") pod "image-registry-5c676dc74-4k7lf" (UID: "d7e081dc-a705-4255-aed8-c20f7090c5b3") : secret "image-registry-tls" not found
Jul 29 22:47:07.697869 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:07.697625    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls podName:ab3523d2-8acf-462d-8458-3b2e6fa74f08 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:23.697619525 +0000 UTC m=+66.255964319 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls") pod "dns-default-25gll" (UID: "ab3523d2-8acf-462d-8458-3b2e6fa74f08") : secret "dns-default-metrics-tls" not found
Jul 29 22:47:09.017242 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:09.017211    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:47:09.079117 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:09.079089    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:47:11.816288 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:11.816252    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:47:11.874199 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:11.874172    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:47:12.118340 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:12.118300    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:47:12.187010 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:12.186970    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:47:18.103664 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:18.103635    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gzpfl"
Jul 29 22:47:18.255456 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:18.255424    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:47:18.306894 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:18.306851    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:47:18.365991 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:18.365935    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:47:23.613704 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.613645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret\") pod \"global-pull-secret-syncer-7pv5p\" (UID: \"8687fc04-c1c6-4547-9787-dc94bfc700c3\") " pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:47:23.616969 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.616950    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 22:47:23.628255 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.628231    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8687fc04-c1c6-4547-9787-dc94bfc700c3-original-pull-secret\") pod \"global-pull-secret-syncer-7pv5p\" (UID: \"8687fc04-c1c6-4547-9787-dc94bfc700c3\") " pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:47:23.714938 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.714913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:47:23.715046 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.714953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sk5qv\" (UID: \"2fe2a7d4-0e16-4f28-9173-d5765b53e33b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:47:23.715046 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.714974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7v2\" (UniqueName: \"kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2\") pod \"network-check-target-82qzb\" (UID: \"71e888ac-8fe5-4706-8af5-0c3227647e96\") " pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:47:23.715114 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:23.715053    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:47:23.715114 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:23.715064    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 22:47:23.715114 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:23.715113    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls podName:ab3523d2-8acf-462d-8458-3b2e6fa74f08 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:55.715097392 +0000 UTC m=+98.273442193 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls") pod "dns-default-25gll" (UID: "ab3523d2-8acf-462d-8458-3b2e6fa74f08") : secret "dns-default-metrics-tls" not found
Jul 29 22:47:23.715211 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.715114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:47:23.715211 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:23.715128    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert podName:2fe2a7d4-0e16-4f28-9173-d5765b53e33b nodeName:}" failed. No retries permitted until 2026-07-29 22:47:55.715121404 +0000 UTC m=+98.273466198 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sk5qv" (UID: "2fe2a7d4-0e16-4f28-9173-d5765b53e33b") : secret "networking-console-plugin-cert" not found
Jul 29 22:47:23.715211 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.715171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs\") pod \"network-metrics-daemon-2ksj4\" (UID: \"cf450310-9f32-4f65-96ea-b4d676ff2df7\") " pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:47:23.715211 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:23.715180    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 22:47:23.715211 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:23.715191    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c676dc74-4k7lf: secret "image-registry-tls" not found
Jul 29 22:47:23.715372 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:23.715223    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls podName:d7e081dc-a705-4255-aed8-c20f7090c5b3 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:55.715211321 +0000 UTC m=+98.273556117 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls") pod "image-registry-5c676dc74-4k7lf" (UID: "d7e081dc-a705-4255-aed8-c20f7090c5b3") : secret "image-registry-tls" not found
Jul 29 22:47:23.718231 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.718217    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 22:47:23.718275 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.718252    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 22:47:23.725845 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:23.725830    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 22:47:23.725890 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:23.725870    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs podName:cf450310-9f32-4f65-96ea-b4d676ff2df7 nodeName:}" failed. No retries permitted until 2026-07-29 22:48:27.725860187 +0000 UTC m=+130.284204981 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs") pod "network-metrics-daemon-2ksj4" (UID: "cf450310-9f32-4f65-96ea-b4d676ff2df7") : secret "metrics-daemon-secret" not found
Jul 29 22:47:23.728253 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.728240    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 22:47:23.738191 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.738167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh7v2\" (UniqueName: \"kubernetes.io/projected/71e888ac-8fe5-4706-8af5-0c3227647e96-kube-api-access-vh7v2\") pod \"network-check-target-82qzb\" (UID: \"71e888ac-8fe5-4706-8af5-0c3227647e96\") " pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:47:23.773670 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.773650    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7pv5p"
Jul 29 22:47:23.791137 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.791118    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-csbtv\""
Jul 29 22:47:23.798657 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.798638    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:47:23.884509 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.884475    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:47:23.902017 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.901969    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7pv5p"]
Jul 29 22:47:23.904703 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:47:23.904679    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8687fc04_c1c6_4547_9787_dc94bfc700c3.slice/crio-88e50170daf9066389c915d5089c1dd356285f0fc4a3154f4b4eee4a19a19f1a WatchSource:0}: Error finding container 88e50170daf9066389c915d5089c1dd356285f0fc4a3154f4b4eee4a19a19f1a: Status 404 returned error can't find the container with id 88e50170daf9066389c915d5089c1dd356285f0fc4a3154f4b4eee4a19a19f1a
Jul 29 22:47:23.928713 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.928693    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-82qzb"]
Jul 29 22:47:23.931090 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:47:23.931069    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e888ac_8fe5_4706_8af5_0c3227647e96.slice/crio-b92b166ede473a3370b30200e4b527eacae6233f5f7a37596810611519c01cf4 WatchSource:0}: Error finding container b92b166ede473a3370b30200e4b527eacae6233f5f7a37596810611519c01cf4: Status 404 returned error can't find the container with id b92b166ede473a3370b30200e4b527eacae6233f5f7a37596810611519c01cf4
Jul 29 22:47:23.983429 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:23.983406    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:47:24.183780 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:24.183700    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-82qzb" event={"ID":"71e888ac-8fe5-4706-8af5-0c3227647e96","Type":"ContainerStarted","Data":"b92b166ede473a3370b30200e4b527eacae6233f5f7a37596810611519c01cf4"}
Jul 29 22:47:24.184654 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:24.184632    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7pv5p" event={"ID":"8687fc04-c1c6-4547-9787-dc94bfc700c3","Type":"ContainerStarted","Data":"88e50170daf9066389c915d5089c1dd356285f0fc4a3154f4b4eee4a19a19f1a"}
Jul 29 22:47:29.198816 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:29.198761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-82qzb" event={"ID":"71e888ac-8fe5-4706-8af5-0c3227647e96","Type":"ContainerStarted","Data":"156f1c4cf7426a471b72a76159b09923a2097e88b56c29dd9144080183884756"}
Jul 29 22:47:29.199326 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:29.198913    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:47:29.200136 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:29.200111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7pv5p" event={"ID":"8687fc04-c1c6-4547-9787-dc94bfc700c3","Type":"ContainerStarted","Data":"f3ec02f6fa91d302a46c9c70fdf231d0390739a8d4b0b0dcc04b59e786785e18"}
Jul 29 22:47:29.214536 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:29.214495    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-82qzb" podStartSLOduration=66.768201153 podStartE2EDuration="1m11.214483117s" totalImagesPullingTime="4.446281964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:47:29.21423451 +0000 UTC m=+71.772579326" watchObservedRunningTime="2026-07-29 22:47:29.214483117 +0000 UTC m=+71.772827958"
Jul 29 22:47:29.230216 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:29.230176    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7pv5p" podStartSLOduration=66.754714975 podStartE2EDuration="1m11.23016473s" totalImagesPullingTime="4.475449755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:47:29.229382152 +0000 UTC m=+71.787726968" watchObservedRunningTime="2026-07-29 22:47:29.23016473 +0000 UTC m=+71.788509546"
Jul 29 22:47:33.837514 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:33.837479    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:47:33.889398 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:33.889368    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:47:45.667054 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:45.667018    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr"]
Jul 29 22:47:45.669538 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:45.669523    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr"
Jul 29 22:47:45.672079 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:45.672059    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 22:47:45.672193 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:45.672111    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 22:47:45.673518 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:45.673498    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 22:47:45.673611 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:45.673507    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 29 22:47:45.679789 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:45.679767    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr"]
Jul 29 22:47:45.767087 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:45.767056    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9wbb\" (UniqueName: \"kubernetes.io/projected/094b7d16-3167-4868-bd8e-313a53e2ed1e-kube-api-access-d9wbb\") pod \"klusterlet-addon-workmgr-6884d49ccd-mh4kr\" (UID: \"094b7d16-3167-4868-bd8e-313a53e2ed1e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr"
Jul 29 22:47:45.767234 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:45.767118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/094b7d16-3167-4868-bd8e-313a53e2ed1e-tmp\") pod \"klusterlet-addon-workmgr-6884d49ccd-mh4kr\" (UID: \"094b7d16-3167-4868-bd8e-313a53e2ed1e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr"
Jul 29 22:47:45.767234 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:45.767198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/094b7d16-3167-4868-bd8e-313a53e2ed1e-klusterlet-config\") pod \"klusterlet-addon-workmgr-6884d49ccd-mh4kr\" (UID: \"094b7d16-3167-4868-bd8e-313a53e2ed1e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr"
Jul 29 22:47:45.867711 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:45.867680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/094b7d16-3167-4868-bd8e-313a53e2ed1e-klusterlet-config\") pod \"klusterlet-addon-workmgr-6884d49ccd-mh4kr\" (UID: \"094b7d16-3167-4868-bd8e-313a53e2ed1e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr"
Jul 29 22:47:45.867834 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:45.867723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9wbb\" (UniqueName: \"kubernetes.io/projected/094b7d16-3167-4868-bd8e-313a53e2ed1e-kube-api-access-d9wbb\") pod \"klusterlet-addon-workmgr-6884d49ccd-mh4kr\" (UID: \"094b7d16-3167-4868-bd8e-313a53e2ed1e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr"
Jul 29 22:47:45.867834 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:45.867763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/094b7d16-3167-4868-bd8e-313a53e2ed1e-tmp\") pod \"klusterlet-addon-workmgr-6884d49ccd-mh4kr\" (UID: \"094b7d16-3167-4868-bd8e-313a53e2ed1e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr"
Jul 29 22:47:45.868091 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:45.868076    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/094b7d16-3167-4868-bd8e-313a53e2ed1e-tmp\") pod \"klusterlet-addon-workmgr-6884d49ccd-mh4kr\" (UID: \"094b7d16-3167-4868-bd8e-313a53e2ed1e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr"
Jul 29 22:47:45.870269 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:45.870247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/094b7d16-3167-4868-bd8e-313a53e2ed1e-klusterlet-config\") pod \"klusterlet-addon-workmgr-6884d49ccd-mh4kr\" (UID: \"094b7d16-3167-4868-bd8e-313a53e2ed1e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr"
Jul 29 22:47:45.876338 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:45.876315    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9wbb\" (UniqueName: \"kubernetes.io/projected/094b7d16-3167-4868-bd8e-313a53e2ed1e-kube-api-access-d9wbb\") pod \"klusterlet-addon-workmgr-6884d49ccd-mh4kr\" (UID: \"094b7d16-3167-4868-bd8e-313a53e2ed1e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr"
Jul 29 22:47:45.978667 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:45.978604    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr"
Jul 29 22:47:46.100123 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:46.100096    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr"]
Jul 29 22:47:46.103194 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:47:46.103166    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094b7d16_3167_4868_bd8e_313a53e2ed1e.slice/crio-174ffcad23148d20b8cbcb28eb01c44e860b7b476583a69dce3dfe7e5435fc74 WatchSource:0}: Error finding container 174ffcad23148d20b8cbcb28eb01c44e860b7b476583a69dce3dfe7e5435fc74: Status 404 returned error can't find the container with id 174ffcad23148d20b8cbcb28eb01c44e860b7b476583a69dce3dfe7e5435fc74
Jul 29 22:47:46.242250 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:46.242188    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr" event={"ID":"094b7d16-3167-4868-bd8e-313a53e2ed1e","Type":"ContainerStarted","Data":"174ffcad23148d20b8cbcb28eb01c44e860b7b476583a69dce3dfe7e5435fc74"}
Jul 29 22:47:50.254382 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:50.254347    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr" event={"ID":"094b7d16-3167-4868-bd8e-313a53e2ed1e","Type":"ContainerStarted","Data":"e05ad1bd8ce5544ccd3e1d86afd5126cc11798bc64c926bd62918e34c3051edb"}
Jul 29 22:47:50.254860 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:50.254522    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr"
Jul 29 22:47:50.256190 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:50.256170    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr"
Jul 29 22:47:50.272308 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:50.272272    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr" podStartSLOduration=1.7026090520000001 podStartE2EDuration="5.272260984s" totalImagesPullingTime="3.569651932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:47:50.271252305 +0000 UTC m=+92.829597122" watchObservedRunningTime="2026-07-29 22:47:50.272260984 +0000 UTC m=+92.830605800"
Jul 29 22:47:55.741130 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:55.741103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:47:55.741434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:55.741150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sk5qv\" (UID: \"2fe2a7d4-0e16-4f28-9173-d5765b53e33b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:47:55.741434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:47:55.741179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:47:55.741434 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:55.741256    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 22:47:55.741434 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:55.741281    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 22:47:55.741434 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:55.741293    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c676dc74-4k7lf: secret "image-registry-tls" not found
Jul 29 22:47:55.741434 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:55.741319    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert podName:2fe2a7d4-0e16-4f28-9173-d5765b53e33b nodeName:}" failed. No retries permitted until 2026-07-29 22:48:59.741302069 +0000 UTC m=+162.299646864 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sk5qv" (UID: "2fe2a7d4-0e16-4f28-9173-d5765b53e33b") : secret "networking-console-plugin-cert" not found
Jul 29 22:47:55.741434 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:55.741333    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls podName:d7e081dc-a705-4255-aed8-c20f7090c5b3 nodeName:}" failed. No retries permitted until 2026-07-29 22:48:59.741326518 +0000 UTC m=+162.299671311 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls") pod "image-registry-5c676dc74-4k7lf" (UID: "d7e081dc-a705-4255-aed8-c20f7090c5b3") : secret "image-registry-tls" not found
Jul 29 22:47:55.741434 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:55.741256    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:47:55.741434 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:47:55.741357    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls podName:ab3523d2-8acf-462d-8458-3b2e6fa74f08 nodeName:}" failed. No retries permitted until 2026-07-29 22:48:59.741350845 +0000 UTC m=+162.299695639 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls") pod "dns-default-25gll" (UID: "ab3523d2-8acf-462d-8458-3b2e6fa74f08") : secret "dns-default-metrics-tls" not found
Jul 29 22:48:00.205207 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:48:00.205181    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-82qzb"
Jul 29 22:48:04.080548 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:48:04.080510    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:48:04.181286 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:48:04.181254    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:48:27.754527 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:48:27.754491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs\") pod \"network-metrics-daemon-2ksj4\" (UID: \"cf450310-9f32-4f65-96ea-b4d676ff2df7\") " pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:48:27.755043 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:48:27.754659    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 22:48:27.755043 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:48:27.754743    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs podName:cf450310-9f32-4f65-96ea-b4d676ff2df7 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:29.754720809 +0000 UTC m=+252.313065607 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs") pod "network-metrics-daemon-2ksj4" (UID: "cf450310-9f32-4f65-96ea-b4d676ff2df7") : secret "metrics-daemon-secret" not found
Jul 29 22:48:54.857099 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:48:54.857056    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5c676dc74-4k7lf" podUID="d7e081dc-a705-4255-aed8-c20f7090c5b3"
Jul 29 22:48:54.864159 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:48:54.864143    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv" podUID="2fe2a7d4-0e16-4f28-9173-d5765b53e33b"
Jul 29 22:48:54.872317 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:48:54.872290    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-25gll" podUID="ab3523d2-8acf-462d-8458-3b2e6fa74f08"
Jul 29 22:48:54.983204 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:48:54.983159    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:48:55.412395 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:48:55.412373    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:48:55.412546 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:48:55.412373    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-25gll"
Jul 29 22:48:55.412546 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:48:55.412375    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:48:59.762609 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:48:59.762578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:48:59.763023 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:48:59.762652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:48:59.763023 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:48:59.762703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sk5qv\" (UID: \"2fe2a7d4-0e16-4f28-9173-d5765b53e33b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:48:59.763023 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:48:59.762735    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 22:48:59.763023 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:48:59.762752    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c676dc74-4k7lf: secret "image-registry-tls" not found
Jul 29 22:48:59.763023 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:48:59.762792    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:48:59.763023 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:48:59.762835    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls podName:d7e081dc-a705-4255-aed8-c20f7090c5b3 nodeName:}" failed. No retries permitted until 2026-07-29 22:51:01.762813553 +0000 UTC m=+284.321158351 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls") pod "image-registry-5c676dc74-4k7lf" (UID: "d7e081dc-a705-4255-aed8-c20f7090c5b3") : secret "image-registry-tls" not found
Jul 29 22:48:59.763023 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:48:59.762860    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls podName:ab3523d2-8acf-462d-8458-3b2e6fa74f08 nodeName:}" failed. No retries permitted until 2026-07-29 22:51:01.762847255 +0000 UTC m=+284.321192054 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls") pod "dns-default-25gll" (UID: "ab3523d2-8acf-462d-8458-3b2e6fa74f08") : secret "dns-default-metrics-tls" not found
Jul 29 22:48:59.763023 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:48:59.762792    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 22:48:59.763023 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:48:59.762894    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert podName:2fe2a7d4-0e16-4f28-9173-d5765b53e33b nodeName:}" failed. No retries permitted until 2026-07-29 22:51:01.762884899 +0000 UTC m=+284.321229695 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sk5qv" (UID: "2fe2a7d4-0e16-4f28-9173-d5765b53e33b") : secret "networking-console-plugin-cert" not found
Jul 29 22:49:08.946451 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:49:08.946407    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:49:32.044399 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:49:32.044310    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:49:50.255460 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:49:50.255414    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr" podUID="094b7d16-3167-4868-bd8e-313a53e2ed1e" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.10:8000/readyz\": dial tcp 10.132.0.10:8000: connect: connection refused"
Jul 29 22:49:50.537386 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:49:50.537308    2581 generic.go:356] "Generic (PLEG): container finished" podID="094b7d16-3167-4868-bd8e-313a53e2ed1e" containerID="e05ad1bd8ce5544ccd3e1d86afd5126cc11798bc64c926bd62918e34c3051edb" exitCode=1
Jul 29 22:49:50.537386 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:49:50.537381    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr" event={"ID":"094b7d16-3167-4868-bd8e-313a53e2ed1e","Type":"ContainerDied","Data":"e05ad1bd8ce5544ccd3e1d86afd5126cc11798bc64c926bd62918e34c3051edb"}
Jul 29 22:49:50.537693 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:49:50.537678    2581 scope.go:117] "RemoveContainer" containerID="e05ad1bd8ce5544ccd3e1d86afd5126cc11798bc64c926bd62918e34c3051edb"
Jul 29 22:49:51.541029 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:49:51.540995    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr" event={"ID":"094b7d16-3167-4868-bd8e-313a53e2ed1e","Type":"ContainerStarted","Data":"476a3c3752fa23e24760e988276d9996d84ef7f906dfcdeaaac1d78e47f9980d"}
Jul 29 22:49:51.541435 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:49:51.541308    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr"
Jul 29 22:49:51.542029 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:49:51.542009    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6884d49ccd-mh4kr"
Jul 29 22:50:19.357907 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:19.357871    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:29.758930 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:29.758893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs\") pod \"network-metrics-daemon-2ksj4\" (UID: \"cf450310-9f32-4f65-96ea-b4d676ff2df7\") " pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:50:29.759354 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:29.758997    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 22:50:29.759354 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:29.759046    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs podName:cf450310-9f32-4f65-96ea-b4d676ff2df7 nodeName:}" failed. No retries permitted until 2026-07-29 22:52:31.759033192 +0000 UTC m=+374.317377987 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs") pod "network-metrics-daemon-2ksj4" (UID: "cf450310-9f32-4f65-96ea-b4d676ff2df7") : secret "metrics-daemon-secret" not found
Jul 29 22:50:44.159219 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.159193    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z9cvg"]
Jul 29 22:50:44.161862 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.161847    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z9cvg"
Jul 29 22:50:44.164756 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.164739    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ckp4z\""
Jul 29 22:50:44.166238 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.166225    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 29 22:50:44.166290 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.166280    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 29 22:50:44.172865 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.172846    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z9cvg"]
Jul 29 22:50:44.257383 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.257362    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq"]
Jul 29 22:50:44.258712 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.258692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c454d\" (UniqueName: \"kubernetes.io/projected/cdc35d2f-cfe4-4a6c-a75c-431cf95edc19-kube-api-access-c454d\") pod \"volume-data-source-validator-549c6d54cf-z9cvg\" (UID: \"cdc35d2f-cfe4-4a6c-a75c-431cf95edc19\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z9cvg"
Jul 29 22:50:44.259973 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.259960    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq"
Jul 29 22:50:44.263093 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.263075    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 29 22:50:44.263195 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.263077    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 29 22:50:44.263412 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.263387    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2xgzl\""
Jul 29 22:50:44.263685 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.263667    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 29 22:50:44.263783 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.263670    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 29 22:50:44.274158 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.274140    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq"]
Jul 29 22:50:44.359420 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.359401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6921abb-a2c9-4b8a-9cf5-6c0d694c2630-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq\" (UID: \"b6921abb-a2c9-4b8a-9cf5-6c0d694c2630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq"
Jul 29 22:50:44.359514 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.359426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6921abb-a2c9-4b8a-9cf5-6c0d694c2630-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq\" (UID: \"b6921abb-a2c9-4b8a-9cf5-6c0d694c2630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq"
Jul 29 22:50:44.359514 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.359459    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz46f\" (UniqueName: \"kubernetes.io/projected/b6921abb-a2c9-4b8a-9cf5-6c0d694c2630-kube-api-access-pz46f\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq\" (UID: \"b6921abb-a2c9-4b8a-9cf5-6c0d694c2630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq"
Jul 29 22:50:44.359594 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.359522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c454d\" (UniqueName: \"kubernetes.io/projected/cdc35d2f-cfe4-4a6c-a75c-431cf95edc19-kube-api-access-c454d\") pod \"volume-data-source-validator-549c6d54cf-z9cvg\" (UID: \"cdc35d2f-cfe4-4a6c-a75c-431cf95edc19\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z9cvg"
Jul 29 22:50:44.381098 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.381075    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c454d\" (UniqueName: \"kubernetes.io/projected/cdc35d2f-cfe4-4a6c-a75c-431cf95edc19-kube-api-access-c454d\") pod \"volume-data-source-validator-549c6d54cf-z9cvg\" (UID: \"cdc35d2f-cfe4-4a6c-a75c-431cf95edc19\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z9cvg"
Jul 29 22:50:44.422418 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.422364    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:44.460259 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.460241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6921abb-a2c9-4b8a-9cf5-6c0d694c2630-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq\" (UID: \"b6921abb-a2c9-4b8a-9cf5-6c0d694c2630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq"
Jul 29 22:50:44.460333 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.460269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6921abb-a2c9-4b8a-9cf5-6c0d694c2630-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq\" (UID: \"b6921abb-a2c9-4b8a-9cf5-6c0d694c2630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq"
Jul 29 22:50:44.460333 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.460308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz46f\" (UniqueName: \"kubernetes.io/projected/b6921abb-a2c9-4b8a-9cf5-6c0d694c2630-kube-api-access-pz46f\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq\" (UID: \"b6921abb-a2c9-4b8a-9cf5-6c0d694c2630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq"
Jul 29 22:50:44.460885 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.460866    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6921abb-a2c9-4b8a-9cf5-6c0d694c2630-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq\" (UID: \"b6921abb-a2c9-4b8a-9cf5-6c0d694c2630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq"
Jul 29 22:50:44.462253 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.462237    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6921abb-a2c9-4b8a-9cf5-6c0d694c2630-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq\" (UID: \"b6921abb-a2c9-4b8a-9cf5-6c0d694c2630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq"
Jul 29 22:50:44.468229 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.468208    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz46f\" (UniqueName: \"kubernetes.io/projected/b6921abb-a2c9-4b8a-9cf5-6c0d694c2630-kube-api-access-pz46f\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq\" (UID: \"b6921abb-a2c9-4b8a-9cf5-6c0d694c2630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq"
Jul 29 22:50:44.471113 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.471092    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z9cvg"
Jul 29 22:50:44.476985 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.476965    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:44.568231 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.568206    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq"
Jul 29 22:50:44.586959 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.586940    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z9cvg"]
Jul 29 22:50:44.589828 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:50:44.589780    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc35d2f_cfe4_4a6c_a75c_431cf95edc19.slice/crio-44d3b1a96e664b311f96c3065ecfc51904de498778c0e213d8e6adb541a5a2bf WatchSource:0}: Error finding container 44d3b1a96e664b311f96c3065ecfc51904de498778c0e213d8e6adb541a5a2bf: Status 404 returned error can't find the container with id 44d3b1a96e664b311f96c3065ecfc51904de498778c0e213d8e6adb541a5a2bf
Jul 29 22:50:44.665237 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.665205    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z9cvg" event={"ID":"cdc35d2f-cfe4-4a6c-a75c-431cf95edc19","Type":"ContainerStarted","Data":"44d3b1a96e664b311f96c3065ecfc51904de498778c0e213d8e6adb541a5a2bf"}
Jul 29 22:50:44.687895 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:44.687792    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq"]
Jul 29 22:50:44.690524 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:50:44.690498    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6921abb_a2c9_4b8a_9cf5_6c0d694c2630.slice/crio-a1d595aeb8066641efe36a1876a50857f144f663c075f4776fee88a088cc8bf9 WatchSource:0}: Error finding container a1d595aeb8066641efe36a1876a50857f144f663c075f4776fee88a088cc8bf9: Status 404 returned error can't find the container with id a1d595aeb8066641efe36a1876a50857f144f663c075f4776fee88a088cc8bf9
Jul 29 22:50:45.670286 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:45.670207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq" event={"ID":"b6921abb-a2c9-4b8a-9cf5-6c0d694c2630","Type":"ContainerStarted","Data":"a1d595aeb8066641efe36a1876a50857f144f663c075f4776fee88a088cc8bf9"}
Jul 29 22:50:46.675973 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:46.675949    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z9cvg" event={"ID":"cdc35d2f-cfe4-4a6c-a75c-431cf95edc19","Type":"ContainerStarted","Data":"1ca1bf5f361517eaf715b59caff7e39b8b86da4152e69846004cd5f21d58cb2c"}
Jul 29 22:50:46.692181 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:46.692132    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z9cvg" podStartSLOduration=1.312078755 podStartE2EDuration="2.692113316s" totalImagesPullingTime="1.380034561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:50:46.690540986 +0000 UTC m=+269.248885803" watchObservedRunningTime="2026-07-29 22:50:46.692113316 +0000 UTC m=+269.250458133"
Jul 29 22:50:46.765105 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:46.765063    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:46.824165 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:46.824133    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:47.682258 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:47.682224    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq" event={"ID":"b6921abb-a2c9-4b8a-9cf5-6c0d694c2630","Type":"ContainerStarted","Data":"d930b9224b768b2d009fc9f9deebdeb442a024275b35c9fd392b4301eb91ec4a"}
Jul 29 22:50:47.700599 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:47.700554    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq" podStartSLOduration=1.717558907 podStartE2EDuration="3.700538802s" totalImagesPullingTime="1.982979895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:50:47.698791321 +0000 UTC m=+270.257136141" watchObservedRunningTime="2026-07-29 22:50:47.700538802 +0000 UTC m=+270.258883618"
Jul 29 22:50:48.268938 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.268906    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:48.278350 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.278327    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-dftrh"]
Jul 29 22:50:48.281306 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.281290    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dftrh"
Jul 29 22:50:48.284008 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.283988    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 29 22:50:48.285262 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.285245    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4n45d\""
Jul 29 22:50:48.285325 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.285263    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 29 22:50:48.291509 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.291489    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-dftrh"]
Jul 29 22:50:48.359053 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.359020    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:48.385028 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.385001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqvs4\" (UniqueName: \"kubernetes.io/projected/66c46109-03d1-4b31-90ac-edb1642d1d04-kube-api-access-vqvs4\") pod \"migrator-6865c9967-dftrh\" (UID: \"66c46109-03d1-4b31-90ac-edb1642d1d04\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dftrh"
Jul 29 22:50:48.460499 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.460469    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:48.486076 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.486054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqvs4\" (UniqueName: \"kubernetes.io/projected/66c46109-03d1-4b31-90ac-edb1642d1d04-kube-api-access-vqvs4\") pod \"migrator-6865c9967-dftrh\" (UID: \"66c46109-03d1-4b31-90ac-edb1642d1d04\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dftrh"
Jul 29 22:50:48.496727 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.496709    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqvs4\" (UniqueName: \"kubernetes.io/projected/66c46109-03d1-4b31-90ac-edb1642d1d04-kube-api-access-vqvs4\") pod \"migrator-6865c9967-dftrh\" (UID: \"66c46109-03d1-4b31-90ac-edb1642d1d04\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dftrh"
Jul 29 22:50:48.589724 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.589702    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dftrh"
Jul 29 22:50:48.711968 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.711942    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-dftrh"]
Jul 29 22:50:48.714906 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:50:48.714884    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c46109_03d1_4b31_90ac_edb1642d1d04.slice/crio-fafbccaae7a576e22267c60dbc8f669644da4c23811661d7f7a0e9326dc99fab WatchSource:0}: Error finding container fafbccaae7a576e22267c60dbc8f669644da4c23811661d7f7a0e9326dc99fab: Status 404 returned error can't find the container with id fafbccaae7a576e22267c60dbc8f669644da4c23811661d7f7a0e9326dc99fab
Jul 29 22:50:48.724826 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.724790    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rr5g7"]
Jul 29 22:50:48.728792 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.728775    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:48.731726 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.731708    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 22:50:48.731842 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.731709    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 22:50:48.732023 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.732001    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 22:50:48.732114 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.732052    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 22:50:48.732114 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.732069    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gfc6t\""
Jul 29 22:50:48.741040 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.741019    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rr5g7"]
Jul 29 22:50:48.776189 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.776166    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:48.790582 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.790561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:48.790644 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.790595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd899\" (UniqueName: \"kubernetes.io/projected/2a191a01-962a-4268-93fc-639ea776d1b6-kube-api-access-pd899\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:48.790709 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.790692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a191a01-962a-4268-93fc-639ea776d1b6-data-volume\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:48.790750 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.790718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a191a01-962a-4268-93fc-639ea776d1b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:48.790785 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.790760    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a191a01-962a-4268-93fc-639ea776d1b6-crio-socket\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:48.891666 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.891614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:48.891666 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.891640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd899\" (UniqueName: \"kubernetes.io/projected/2a191a01-962a-4268-93fc-639ea776d1b6-kube-api-access-pd899\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:48.891838 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.891681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a191a01-962a-4268-93fc-639ea776d1b6-data-volume\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:48.891838 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.891699    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a191a01-962a-4268-93fc-639ea776d1b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:48.891838 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.891733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a191a01-962a-4268-93fc-639ea776d1b6-crio-socket\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:48.891838 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:48.891736    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:48.891838 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:48.891781    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls podName:2a191a01-962a-4268-93fc-639ea776d1b6 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:49.391767466 +0000 UTC m=+271.950112263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rr5g7" (UID: "2a191a01-962a-4268-93fc-639ea776d1b6") : secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:48.891838 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.891838    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a191a01-962a-4268-93fc-639ea776d1b6-crio-socket\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:48.892063 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:48.891867    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 22:50:48.892063 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:48.891918    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a191a01-962a-4268-93fc-639ea776d1b6-kube-rbac-proxy-cm podName:2a191a01-962a-4268-93fc-639ea776d1b6 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:49.391905724 +0000 UTC m=+271.950250534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a191a01-962a-4268-93fc-639ea776d1b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rr5g7" (UID: "2a191a01-962a-4268-93fc-639ea776d1b6") : configmap "kube-rbac-proxy" not found
Jul 29 22:50:48.892063 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.891955    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a191a01-962a-4268-93fc-639ea776d1b6-data-volume\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:48.899685 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:48.899667    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd899\" (UniqueName: \"kubernetes.io/projected/2a191a01-962a-4268-93fc-639ea776d1b6-kube-api-access-pd899\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:49.044541 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:49.044511    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:49.099822 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:49.099786    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:49.393960 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:49.393936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:49.394091 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:49.393987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a191a01-962a-4268-93fc-639ea776d1b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:49.394091 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:49.394044    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 22:50:49.394091 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:49.394071    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:49.394185 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:49.394094    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a191a01-962a-4268-93fc-639ea776d1b6-kube-rbac-proxy-cm podName:2a191a01-962a-4268-93fc-639ea776d1b6 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:50.39407699 +0000 UTC m=+272.952421801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a191a01-962a-4268-93fc-639ea776d1b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rr5g7" (UID: "2a191a01-962a-4268-93fc-639ea776d1b6") : configmap "kube-rbac-proxy" not found
Jul 29 22:50:49.394185 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:49.394115    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls podName:2a191a01-962a-4268-93fc-639ea776d1b6 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:50.394103563 +0000 UTC m=+272.952448359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rr5g7" (UID: "2a191a01-962a-4268-93fc-639ea776d1b6") : secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:49.667705 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:49.667613    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:49.694130 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:49.694097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dftrh" event={"ID":"66c46109-03d1-4b31-90ac-edb1642d1d04","Type":"ContainerStarted","Data":"fafbccaae7a576e22267c60dbc8f669644da4c23811661d7f7a0e9326dc99fab"}
Jul 29 22:50:49.769896 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:49.769863    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:50.401829 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:50.401789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:50.401983 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:50.401860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a191a01-962a-4268-93fc-639ea776d1b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:50.401983 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:50.401925    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 22:50:50.401983 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:50.401942    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:50.401983 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:50.401968    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a191a01-962a-4268-93fc-639ea776d1b6-kube-rbac-proxy-cm podName:2a191a01-962a-4268-93fc-639ea776d1b6 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:52.401955908 +0000 UTC m=+274.960300701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a191a01-962a-4268-93fc-639ea776d1b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rr5g7" (UID: "2a191a01-962a-4268-93fc-639ea776d1b6") : configmap "kube-rbac-proxy" not found
Jul 29 22:50:50.402134 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:50.401996    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls podName:2a191a01-962a-4268-93fc-639ea776d1b6 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:52.401980124 +0000 UTC m=+274.960324933 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rr5g7" (UID: "2a191a01-962a-4268-93fc-639ea776d1b6") : secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:50.698548 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:50.698480    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dftrh" event={"ID":"66c46109-03d1-4b31-90ac-edb1642d1d04","Type":"ContainerStarted","Data":"1299fd6633da310b9867336e9bebae81cf06637a347d78438a794cf96bc21d73"}
Jul 29 22:50:50.698548 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:50.698526    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dftrh" event={"ID":"66c46109-03d1-4b31-90ac-edb1642d1d04","Type":"ContainerStarted","Data":"9c29d1a5113a6291517dffaa6688cf162d8978fcde82a75a55f843ef3232352e"}
Jul 29 22:50:50.715086 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:50.715040    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dftrh" podStartSLOduration=1.3493671950000001 podStartE2EDuration="2.715026234s" totalImagesPullingTime="1.365659039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:50:50.714291184 +0000 UTC m=+273.272636001" watchObservedRunningTime="2026-07-29 22:50:50.715026234 +0000 UTC m=+273.273371051"
Jul 29 22:50:50.788579 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:50.788549    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:51.281985 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.281951    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-xqpb8"]
Jul 29 22:50:51.285151 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.285131    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-xqpb8"
Jul 29 22:50:51.288380 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.288350    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 29 22:50:51.288380 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.288350    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 29 22:50:51.289719 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.289698    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 29 22:50:51.289719 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.289717    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fzk8g\""
Jul 29 22:50:51.289913 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.289780    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 29 22:50:51.294940 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.294911    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-xqpb8"]
Jul 29 22:50:51.308704 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.308675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7253c3b3-de17-4c73-be64-834052ce468d-signing-cabundle\") pod \"service-ca-658d7b7c78-xqpb8\" (UID: \"7253c3b3-de17-4c73-be64-834052ce468d\") " pod="openshift-service-ca/service-ca-658d7b7c78-xqpb8"
Jul 29 22:50:51.308822 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.308718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnp2x\" (UniqueName: \"kubernetes.io/projected/7253c3b3-de17-4c73-be64-834052ce468d-kube-api-access-qnp2x\") pod \"service-ca-658d7b7c78-xqpb8\" (UID: \"7253c3b3-de17-4c73-be64-834052ce468d\") " pod="openshift-service-ca/service-ca-658d7b7c78-xqpb8"
Jul 29 22:50:51.308822 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.308793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7253c3b3-de17-4c73-be64-834052ce468d-signing-key\") pod \"service-ca-658d7b7c78-xqpb8\" (UID: \"7253c3b3-de17-4c73-be64-834052ce468d\") " pod="openshift-service-ca/service-ca-658d7b7c78-xqpb8"
Jul 29 22:50:51.328046 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.328016    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:51.395053 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.395026    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:51.409838 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.409818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7253c3b3-de17-4c73-be64-834052ce468d-signing-key\") pod \"service-ca-658d7b7c78-xqpb8\" (UID: \"7253c3b3-de17-4c73-be64-834052ce468d\") " pod="openshift-service-ca/service-ca-658d7b7c78-xqpb8"
Jul 29 22:50:51.409909 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.409877    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7253c3b3-de17-4c73-be64-834052ce468d-signing-cabundle\") pod \"service-ca-658d7b7c78-xqpb8\" (UID: \"7253c3b3-de17-4c73-be64-834052ce468d\") " pod="openshift-service-ca/service-ca-658d7b7c78-xqpb8"
Jul 29 22:50:51.409947 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.409930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnp2x\" (UniqueName: \"kubernetes.io/projected/7253c3b3-de17-4c73-be64-834052ce468d-kube-api-access-qnp2x\") pod \"service-ca-658d7b7c78-xqpb8\" (UID: \"7253c3b3-de17-4c73-be64-834052ce468d\") " pod="openshift-service-ca/service-ca-658d7b7c78-xqpb8"
Jul 29 22:50:51.410534 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.410516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7253c3b3-de17-4c73-be64-834052ce468d-signing-cabundle\") pod \"service-ca-658d7b7c78-xqpb8\" (UID: \"7253c3b3-de17-4c73-be64-834052ce468d\") " pod="openshift-service-ca/service-ca-658d7b7c78-xqpb8"
Jul 29 22:50:51.412101 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.412083    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7253c3b3-de17-4c73-be64-834052ce468d-signing-key\") pod \"service-ca-658d7b7c78-xqpb8\" (UID: \"7253c3b3-de17-4c73-be64-834052ce468d\") " pod="openshift-service-ca/service-ca-658d7b7c78-xqpb8"
Jul 29 22:50:51.419128 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.419107    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnp2x\" (UniqueName: \"kubernetes.io/projected/7253c3b3-de17-4c73-be64-834052ce468d-kube-api-access-qnp2x\") pod \"service-ca-658d7b7c78-xqpb8\" (UID: \"7253c3b3-de17-4c73-be64-834052ce468d\") " pod="openshift-service-ca/service-ca-658d7b7c78-xqpb8"
Jul 29 22:50:51.595784 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.595763    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-xqpb8"
Jul 29 22:50:51.714507 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.714481    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-xqpb8"]
Jul 29 22:50:51.717380 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:50:51.717358    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7253c3b3_de17_4c73_be64_834052ce468d.slice/crio-22bfa70bcd19b005a36d963000dcf299ac87a8c05e558360715511ada145b6c3 WatchSource:0}: Error finding container 22bfa70bcd19b005a36d963000dcf299ac87a8c05e558360715511ada145b6c3: Status 404 returned error can't find the container with id 22bfa70bcd19b005a36d963000dcf299ac87a8c05e558360715511ada145b6c3
Jul 29 22:50:51.740758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:51.740738    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-whv2s_0ad705ff-e172-497a-a972-108f30e7be6b/dns-node-resolver/0.log"
Jul 29 22:50:52.341275 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:52.341249    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8tftg_bb349f88-27d7-4900-9d7c-d0e7916098ab/node-ca/0.log"
Jul 29 22:50:52.416265 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:52.416231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:52.416419 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:52.416307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a191a01-962a-4268-93fc-639ea776d1b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:52.416419 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:52.416375    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 22:50:52.416419 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:52.416394    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:52.416541 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:52.416438    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls podName:2a191a01-962a-4268-93fc-639ea776d1b6 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:56.41642142 +0000 UTC m=+278.974766231 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rr5g7" (UID: "2a191a01-962a-4268-93fc-639ea776d1b6") : secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:52.416541 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:52.416454    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a191a01-962a-4268-93fc-639ea776d1b6-kube-rbac-proxy-cm podName:2a191a01-962a-4268-93fc-639ea776d1b6 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:56.416445703 +0000 UTC m=+278.974790505 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a191a01-962a-4268-93fc-639ea776d1b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rr5g7" (UID: "2a191a01-962a-4268-93fc-639ea776d1b6") : configmap "kube-rbac-proxy" not found
Jul 29 22:50:52.669244 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:52.669165    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:52.706285 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:52.706253    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-xqpb8" event={"ID":"7253c3b3-de17-4c73-be64-834052ce468d","Type":"ContainerStarted","Data":"22bfa70bcd19b005a36d963000dcf299ac87a8c05e558360715511ada145b6c3"}
Jul 29 22:50:53.341302 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:53.341237    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hlgwc_81d4f0dc-2d3a-4523-83d4-de99adcf5e03/serve-healthcheck-canary/0.log"
Jul 29 22:50:53.710903 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:53.710825    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-xqpb8" event={"ID":"7253c3b3-de17-4c73-be64-834052ce468d","Type":"ContainerStarted","Data":"e9d358470dd0dda9f3ad8a78a731739c0227aca4ac57ecce1a548c20d74542cc"}
Jul 29 22:50:53.727558 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:53.727514    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-xqpb8" podStartSLOduration=0.960147509 podStartE2EDuration="2.727501601s" totalImagesPullingTime="1.767354092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:50:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:50:53.726901858 +0000 UTC m=+276.285246675" watchObservedRunningTime="2026-07-29 22:50:53.727501601 +0000 UTC m=+276.285846417"
Jul 29 22:50:53.740811 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:53.740776    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-dftrh_66c46109-03d1-4b31-90ac-edb1642d1d04/migrator/0.log"
Jul 29 22:50:53.941694 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:53.941662    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-dftrh_66c46109-03d1-4b31-90ac-edb1642d1d04/graceful-termination/0.log"
Jul 29 22:50:54.143355 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:54.143328    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq_b6921abb-a2c9-4b8a-9cf5-6c0d694c2630/kube-storage-version-migrator-operator/0.log"
Jul 29 22:50:55.126865 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:55.126834    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:55.254123 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:55.254082    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:55.354688 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:55.354656    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:55.461149 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:55.461071    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:55.616787 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:55.616753    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:55.836435 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:55.836403    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:56.050051 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:56.050025    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:56.448861 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:56.448821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a191a01-962a-4268-93fc-639ea776d1b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:56.449202 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:56.448900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:50:56.449202 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:56.448940    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 22:50:56.449202 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:56.448980    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:56.449202 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:56.449006    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a191a01-962a-4268-93fc-639ea776d1b6-kube-rbac-proxy-cm podName:2a191a01-962a-4268-93fc-639ea776d1b6 nodeName:}" failed. No retries permitted until 2026-07-29 22:51:04.448989954 +0000 UTC m=+287.007334769 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a191a01-962a-4268-93fc-639ea776d1b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rr5g7" (UID: "2a191a01-962a-4268-93fc-639ea776d1b6") : configmap "kube-rbac-proxy" not found
Jul 29 22:50:56.449202 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:56.449023    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls podName:2a191a01-962a-4268-93fc-639ea776d1b6 nodeName:}" failed. No retries permitted until 2026-07-29 22:51:04.449012559 +0000 UTC m=+287.007357357 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rr5g7" (UID: "2a191a01-962a-4268-93fc-639ea776d1b6") : secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:56.830944 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:56.830863    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:57.239420 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:57.239389    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:57.428543 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:57.428510    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:57.621338 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:57.621307    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:57.824465 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:57.824434    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:58.413368 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:58.413333    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv" podUID="2fe2a7d4-0e16-4f28-9173-d5765b53e33b"
Jul 29 22:50:58.413368 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:58.413353    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5c676dc74-4k7lf" podUID="d7e081dc-a705-4255-aed8-c20f7090c5b3"
Jul 29 22:50:58.413561 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:50:58.413356    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-25gll" podUID="ab3523d2-8acf-462d-8458-3b2e6fa74f08"
Jul 29 22:50:58.726915 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:58.726836    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-25gll"
Jul 29 22:50:58.727274 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:58.726846    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:50:58.727274 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:58.726846    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:50:58.941478 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:58.941440    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:59.042160 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:59.042088    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:59.103857 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:59.103826    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:59.228523 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:59.228499    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:59.421756 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:59.421736    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:59.622978 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:59.622922    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:50:59.821456 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:50:59.821375    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:00.823275 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:00.823241    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:01.021511 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:01.021474    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:01.421634 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:01.421603    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:01.519729 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:01.519701    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:01.579732 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:01.579711    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:01.633504 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:01.633481    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:01.789567 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:01.789514    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:51:01.789567 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:01.789555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:51:01.789781 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:01.789765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sk5qv\" (UID: \"2fe2a7d4-0e16-4f28-9173-d5765b53e33b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:51:01.789879 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:51:01.789855    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 22:51:01.789936 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:51:01.789912    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert podName:2fe2a7d4-0e16-4f28-9173-d5765b53e33b nodeName:}" failed. No retries permitted until 2026-07-29 22:53:03.789899101 +0000 UTC m=+406.348243900 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sk5qv" (UID: "2fe2a7d4-0e16-4f28-9173-d5765b53e33b") : secret "networking-console-plugin-cert" not found
Jul 29 22:51:01.791770 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:01.791744    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab3523d2-8acf-462d-8458-3b2e6fa74f08-metrics-tls\") pod \"dns-default-25gll\" (UID: \"ab3523d2-8acf-462d-8458-3b2e6fa74f08\") " pod="openshift-dns/dns-default-25gll"
Jul 29 22:51:01.791859 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:01.791851    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls\") pod \"image-registry-5c676dc74-4k7lf\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") " pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:51:02.022353 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:02.022317    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:02.030240 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:02.030220    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jtttq\""
Jul 29 22:51:02.031481 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:02.031467    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vpnhm\""
Jul 29 22:51:02.037773 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:02.037760    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:51:02.037840 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:02.037787    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-25gll"
Jul 29 22:51:02.179611 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:02.179583    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c676dc74-4k7lf"]
Jul 29 22:51:02.183258 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:51:02.183230    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e081dc_a705_4255_aed8_c20f7090c5b3.slice/crio-849a5d1bab6a6f9b7201c62f3c0d4e0b75000b3e202f08f7f32818c49c297e87 WatchSource:0}: Error finding container 849a5d1bab6a6f9b7201c62f3c0d4e0b75000b3e202f08f7f32818c49c297e87: Status 404 returned error can't find the container with id 849a5d1bab6a6f9b7201c62f3c0d4e0b75000b3e202f08f7f32818c49c297e87
Jul 29 22:51:02.191714 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:02.191694    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-25gll"]
Jul 29 22:51:02.195432 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:51:02.195408    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3523d2_8acf_462d_8458_3b2e6fa74f08.slice/crio-f86bfaf1a61590a9957f5cdb5b135885bf44ba42ad42f211527f14aa110794e6 WatchSource:0}: Error finding container f86bfaf1a61590a9957f5cdb5b135885bf44ba42ad42f211527f14aa110794e6: Status 404 returned error can't find the container with id f86bfaf1a61590a9957f5cdb5b135885bf44ba42ad42f211527f14aa110794e6
Jul 29 22:51:02.740195 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:02.740160    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-25gll" event={"ID":"ab3523d2-8acf-462d-8458-3b2e6fa74f08","Type":"ContainerStarted","Data":"f86bfaf1a61590a9957f5cdb5b135885bf44ba42ad42f211527f14aa110794e6"}
Jul 29 22:51:02.741505 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:02.741484    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c676dc74-4k7lf" event={"ID":"d7e081dc-a705-4255-aed8-c20f7090c5b3","Type":"ContainerStarted","Data":"a769bb6e937a23456737b247e48ba37eba3d6885ac8ce19e84b14b1d52764bef"}
Jul 29 22:51:02.741587 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:02.741519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c676dc74-4k7lf" event={"ID":"d7e081dc-a705-4255-aed8-c20f7090c5b3","Type":"ContainerStarted","Data":"849a5d1bab6a6f9b7201c62f3c0d4e0b75000b3e202f08f7f32818c49c297e87"}
Jul 29 22:51:02.741649 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:02.741622    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:51:02.761333 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:02.761270    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c676dc74-4k7lf" podStartSLOduration=262.76125311 podStartE2EDuration="4m22.76125311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:02.761251897 +0000 UTC m=+285.319596714" watchObservedRunningTime="2026-07-29 22:51:02.76125311 +0000 UTC m=+285.319597934"
Jul 29 22:51:02.981144 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:02.981104    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:03.104423 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:03.104387    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:03.224825 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:03.224709    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:03.424089 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:03.424008    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:03.623148 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:03.623115    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:04.021338 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:04.021311    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:04.517410 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:04.517374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:51:04.517793 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:04.517439    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a191a01-962a-4268-93fc-639ea776d1b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:51:04.517793 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:51:04.517538    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 22:51:04.517793 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:51:04.517607    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls podName:2a191a01-962a-4268-93fc-639ea776d1b6 nodeName:}" failed. No retries permitted until 2026-07-29 22:51:20.517589901 +0000 UTC m=+303.075934696 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rr5g7" (UID: "2a191a01-962a-4268-93fc-639ea776d1b6") : secret "insights-runtime-extractor-tls" not found
Jul 29 22:51:04.518057 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:04.518037    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a191a01-962a-4268-93fc-639ea776d1b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:51:04.625500 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:04.625469    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:04.749527 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:04.749499    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-25gll" event={"ID":"ab3523d2-8acf-462d-8458-3b2e6fa74f08","Type":"ContainerStarted","Data":"c3190d7e2ca8be1413f58c68ab3b1fc26f48b2682a58d1729d846dc8cef2b02d"}
Jul 29 22:51:04.749637 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:04.749578    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-25gll" event={"ID":"ab3523d2-8acf-462d-8458-3b2e6fa74f08","Type":"ContainerStarted","Data":"b310d9cd3f5237958fec6326be397e564c65b4124d5e343ae9cf984d70f425bb"}
Jul 29 22:51:04.749682 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:04.749635    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-25gll"
Jul 29 22:51:04.768369 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:04.768282    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-25gll" podStartSLOduration=251.954194199 podStartE2EDuration="4m13.768269894s" totalImagesPullingTime="1.814075695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:04.766536135 +0000 UTC m=+287.324880945" watchObservedRunningTime="2026-07-29 22:51:04.768269894 +0000 UTC m=+287.326614711"
Jul 29 22:51:05.216919 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:05.216884    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:05.424140 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:05.424095    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:05.625103 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:05.625075    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:05.818952 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:05.818929    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:10.610992 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:10.610953    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:10.712247 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:10.712221    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:11.946965 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:51:11.946916    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-2ksj4" podUID="cf450310-9f32-4f65-96ea-b4d676ff2df7"
Jul 29 22:51:13.080030 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:13.079993    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:13.177937 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:13.177903    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:14.755006 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:14.754972    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-25gll"
Jul 29 22:51:17.822320 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:17.822295    2581 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 22:51:18.675112 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:18.675076    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:18.774354 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:18.774312    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:18.835512 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:18.835483    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-98g9k"]
Jul 29 22:51:18.837248 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:18.837233    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-98g9k"
Jul 29 22:51:18.839986 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:18.839959    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 29 22:51:18.840113 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:18.840018    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rdjmz\""
Jul 29 22:51:18.841054 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:18.840994    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:18.846406 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:18.846388    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-98g9k"]
Jul 29 22:51:18.914092 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:18.914070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0cbd4a90-8b07-483b-9fc6-e3ac61d55df9-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-98g9k\" (UID: \"0cbd4a90-8b07-483b-9fc6-e3ac61d55df9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-98g9k"
Jul 29 22:51:19.014442 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:19.014383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0cbd4a90-8b07-483b-9fc6-e3ac61d55df9-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-98g9k\" (UID: \"0cbd4a90-8b07-483b-9fc6-e3ac61d55df9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-98g9k"
Jul 29 22:51:19.014524 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:51:19.014513    2581 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 29 22:51:19.014574 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:51:19.014564    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cbd4a90-8b07-483b-9fc6-e3ac61d55df9-tls-certificates podName:0cbd4a90-8b07-483b-9fc6-e3ac61d55df9 nodeName:}" failed. No retries permitted until 2026-07-29 22:51:19.514551534 +0000 UTC m=+302.072896328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/0cbd4a90-8b07-483b-9fc6-e3ac61d55df9-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-98g9k" (UID: "0cbd4a90-8b07-483b-9fc6-e3ac61d55df9") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 29 22:51:19.464337 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:19.464305    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:19.518038 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:19.518014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0cbd4a90-8b07-483b-9fc6-e3ac61d55df9-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-98g9k\" (UID: \"0cbd4a90-8b07-483b-9fc6-e3ac61d55df9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-98g9k"
Jul 29 22:51:19.520274 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:19.520248    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0cbd4a90-8b07-483b-9fc6-e3ac61d55df9-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-98g9k\" (UID: \"0cbd4a90-8b07-483b-9fc6-e3ac61d55df9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-98g9k"
Jul 29 22:51:19.746549 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:19.746473    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-98g9k"
Jul 29 22:51:19.864408 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:19.864346    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-98g9k"]
Jul 29 22:51:19.866716 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:51:19.866693    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbd4a90_8b07_483b_9fc6_e3ac61d55df9.slice/crio-5d2cc5eb55ce09158c11a387e7b679199b007d17e0d5803cf8fe54082b56c4dd WatchSource:0}: Error finding container 5d2cc5eb55ce09158c11a387e7b679199b007d17e0d5803cf8fe54082b56c4dd: Status 404 returned error can't find the container with id 5d2cc5eb55ce09158c11a387e7b679199b007d17e0d5803cf8fe54082b56c4dd
Jul 29 22:51:19.868719 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:19.868700    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 22:51:20.524661 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:20.524630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:51:20.527108 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:20.527080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a191a01-962a-4268-93fc-639ea776d1b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rr5g7\" (UID: \"2a191a01-962a-4268-93fc-639ea776d1b6\") " pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:51:20.540811 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:20.540780    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gfc6t\""
Jul 29 22:51:20.548400 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:20.548377    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rr5g7"
Jul 29 22:51:20.685648 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:20.685609    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rr5g7"]
Jul 29 22:51:20.688659 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:51:20.688629    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a191a01_962a_4268_93fc_639ea776d1b6.slice/crio-1d33250426312315d2f4024336a6877feb79f8da0e01e6e1a7565b1e98563eb5 WatchSource:0}: Error finding container 1d33250426312315d2f4024336a6877feb79f8da0e01e6e1a7565b1e98563eb5: Status 404 returned error can't find the container with id 1d33250426312315d2f4024336a6877feb79f8da0e01e6e1a7565b1e98563eb5
Jul 29 22:51:20.802999 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:20.802926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-98g9k" event={"ID":"0cbd4a90-8b07-483b-9fc6-e3ac61d55df9","Type":"ContainerStarted","Data":"5d2cc5eb55ce09158c11a387e7b679199b007d17e0d5803cf8fe54082b56c4dd"}
Jul 29 22:51:20.804193 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:20.804167    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rr5g7" event={"ID":"2a191a01-962a-4268-93fc-639ea776d1b6","Type":"ContainerStarted","Data":"1d33250426312315d2f4024336a6877feb79f8da0e01e6e1a7565b1e98563eb5"}
Jul 29 22:51:21.809080 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:21.809043    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-98g9k" event={"ID":"0cbd4a90-8b07-483b-9fc6-e3ac61d55df9","Type":"ContainerStarted","Data":"b5cf6094810eff48ca21b2cd4bcfafa67c43cf51084a6f96bac758a40c56ee8f"}
Jul 29 22:51:21.809540 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:21.809276    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-98g9k"
Jul 29 22:51:21.814903 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:21.814881    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-98g9k"
Jul 29 22:51:21.827629 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:21.827586    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-98g9k" podStartSLOduration=2.703277013 podStartE2EDuration="3.827572994s" totalImagesPullingTime="1.124295981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:21.826664724 +0000 UTC m=+304.385009544" watchObservedRunningTime="2026-07-29 22:51:21.827572994 +0000 UTC m=+304.385917811"
Jul 29 22:51:22.042386 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:22.042348    2581 patch_prober.go:28] interesting pod/image-registry-5c676dc74-4k7lf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 29 22:51:22.042532 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:22.042402    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5c676dc74-4k7lf" podUID="d7e081dc-a705-4255-aed8-c20f7090c5b3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 22:51:22.875198 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:22.875162    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:22.928028 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:22.927997    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-vq8rs"]
Jul 29 22:51:22.931287 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:22.931264    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs"
Jul 29 22:51:22.936225 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:22.936200    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 29 22:51:22.936384 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:22.936231    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 22:51:22.936384 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:22.936259    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 22:51:22.936384 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:22.936210    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7zgtl\""
Jul 29 22:51:22.936575 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:22.936525    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 22:51:22.936575 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:22.936545    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 29 22:51:22.939381 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:22.939356    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-vq8rs"]
Jul 29 22:51:23.047198 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:23.047165    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-vq8rs\" (UID: \"7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs"
Jul 29 22:51:23.047367 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:23.047213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-vq8rs\" (UID: \"7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs"
Jul 29 22:51:23.047367 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:23.047291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-vq8rs\" (UID: \"7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs"
Jul 29 22:51:23.047478 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:23.047385    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqjr2\" (UniqueName: \"kubernetes.io/projected/7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8-kube-api-access-jqjr2\") pod \"prometheus-operator-84d69f794d-vq8rs\" (UID: \"7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs"
Jul 29 22:51:23.148528 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:23.148447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-vq8rs\" (UID: \"7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs"
Jul 29 22:51:23.148528 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:23.148493    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-vq8rs\" (UID: \"7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs"
Jul 29 22:51:23.148739 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:23.148687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-vq8rs\" (UID: \"7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs"
Jul 29 22:51:23.148819 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:23.148784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqjr2\" (UniqueName: \"kubernetes.io/projected/7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8-kube-api-access-jqjr2\") pod \"prometheus-operator-84d69f794d-vq8rs\" (UID: \"7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs"
Jul 29 22:51:23.148879 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:51:23.148849    2581 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 29 22:51:23.148930 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:51:23.148920    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8-prometheus-operator-tls podName:7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8 nodeName:}" failed. No retries permitted until 2026-07-29 22:51:23.648900538 +0000 UTC m=+306.207245332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-vq8rs" (UID: "7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8") : secret "prometheus-operator-tls" not found
Jul 29 22:51:23.149202 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:23.149182    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-vq8rs\" (UID: \"7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs"
Jul 29 22:51:23.151278 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:23.151238    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-vq8rs\" (UID: \"7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs"
Jul 29 22:51:23.157900 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:23.157876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqjr2\" (UniqueName: \"kubernetes.io/projected/7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8-kube-api-access-jqjr2\") pod \"prometheus-operator-84d69f794d-vq8rs\" (UID: \"7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs"
Jul 29 22:51:23.653754 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:23.653716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-vq8rs\" (UID: \"7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs"
Jul 29 22:51:23.656430 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:23.656403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-vq8rs\" (UID: \"7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs"
Jul 29 22:51:23.750147 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:23.750086    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:51:23.824161 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:23.824115    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:23.841849 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:23.841823    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs"
Jul 29 22:51:23.925051 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:23.924966    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:24.025993 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:24.025949    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:24.084980 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:24.084940    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:24.946982 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:24.946943    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:51:25.565216 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:25.565185    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:25.603926 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:25.603897    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-vq8rs"]
Jul 29 22:51:25.607727 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:51:25.607699    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8631d3_6cf9_4ca9_8c11_73dcb9b8b1d8.slice/crio-c8bb9406f65d6f11bfb97a3c703f7a60f9ff4e22f585cee439f2aba4e26c96a0 WatchSource:0}: Error finding container c8bb9406f65d6f11bfb97a3c703f7a60f9ff4e22f585cee439f2aba4e26c96a0: Status 404 returned error can't find the container with id c8bb9406f65d6f11bfb97a3c703f7a60f9ff4e22f585cee439f2aba4e26c96a0
Jul 29 22:51:25.665454 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:25.665425    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:25.826671 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:25.826578    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rr5g7" event={"ID":"2a191a01-962a-4268-93fc-639ea776d1b6","Type":"ContainerStarted","Data":"484add31896a0141f2d8f3bba3d925fe801201a9bda6da46019960c8a051779e"}
Jul 29 22:51:25.827632 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:25.827610    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs" event={"ID":"7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8","Type":"ContainerStarted","Data":"c8bb9406f65d6f11bfb97a3c703f7a60f9ff4e22f585cee439f2aba4e26c96a0"}
Jul 29 22:51:29.615299 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:29.615263    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:29.846613 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:29.846557    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs" event={"ID":"7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8","Type":"ContainerStarted","Data":"cd032c7a06390356be055e45b3b88e8119313d825d738952e55d9fa58f8724ee"}
Jul 29 22:51:29.846868 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:29.846625    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs" event={"ID":"7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8","Type":"ContainerStarted","Data":"6cb1724a3eea27eb9e4ebe1d0d9ce5b0276d7d83ece3204aab833a042bb58c05"}
Jul 29 22:51:29.848815 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:29.848770    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rr5g7" event={"ID":"2a191a01-962a-4268-93fc-639ea776d1b6","Type":"ContainerStarted","Data":"f65927a6c6da0f00a8bacc48a2d7c3d8e90d4011702991c06b542fc23e75a26e"}
Jul 29 22:51:29.865633 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:29.865561    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-vq8rs" podStartSLOduration=4.185424369 podStartE2EDuration="7.865548263s" totalImagesPullingTime="3.680123894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:51:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:29.863943841 +0000 UTC m=+312.422288659" watchObservedRunningTime="2026-07-29 22:51:29.865548263 +0000 UTC m=+312.423893058"
Jul 29 22:51:30.128534 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:30.128447    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:30.224878 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:30.224839    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:30.320979 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:30.320943    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:30.423737 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:30.423643    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:30.793671 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:30.793575    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:30.900059 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:30.900021    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:31.101688 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.101652    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:31.252628 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.252595    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:31.355567 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.355426    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:31.517653 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.517622    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w"]
Jul 29 22:51:31.523211 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.523187    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w"
Jul 29 22:51:31.534007 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.533680    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 29 22:51:31.534007 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.533715    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2pwsl\""
Jul 29 22:51:31.534193 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.534175    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 29 22:51:31.542324 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.542055    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w"]
Jul 29 22:51:31.556730 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.556697    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:31.588703 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.588670    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jrx9p"]
Jul 29 22:51:31.593828 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.593806    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.599066 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.599044    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 22:51:31.599208 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.599195    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 22:51:31.599428 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.599412    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xxcw6\""
Jul 29 22:51:31.601043 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.601025    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 22:51:31.621555 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.621445    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc5cd\" (UniqueName: \"kubernetes.io/projected/94e94917-7911-4a16-a229-367716e81a26-kube-api-access-wc5cd\") pod \"openshift-state-metrics-b8f49f5c6-6mt7w\" (UID: \"94e94917-7911-4a16-a229-367716e81a26\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w"
Jul 29 22:51:31.621654 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.621563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/94e94917-7911-4a16-a229-367716e81a26-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-6mt7w\" (UID: \"94e94917-7911-4a16-a229-367716e81a26\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w"
Jul 29 22:51:31.621707 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.621651    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94e94917-7911-4a16-a229-367716e81a26-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-6mt7w\" (UID: \"94e94917-7911-4a16-a229-367716e81a26\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w"
Jul 29 22:51:31.621707 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.621692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94e94917-7911-4a16-a229-367716e81a26-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-6mt7w\" (UID: \"94e94917-7911-4a16-a229-367716e81a26\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w"
Jul 29 22:51:31.660691 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.660660    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:31.722596 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.722570    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3907412a-9084-46b6-b25c-eaf316459f5e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.722724 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.722618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94e94917-7911-4a16-a229-367716e81a26-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-6mt7w\" (UID: \"94e94917-7911-4a16-a229-367716e81a26\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w"
Jul 29 22:51:31.722724 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.722682    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94e94917-7911-4a16-a229-367716e81a26-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-6mt7w\" (UID: \"94e94917-7911-4a16-a229-367716e81a26\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w"
Jul 29 22:51:31.722862 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.722750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3907412a-9084-46b6-b25c-eaf316459f5e-sys\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.722862 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.722776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3907412a-9084-46b6-b25c-eaf316459f5e-node-exporter-wtmp\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.722862 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.722826    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3907412a-9084-46b6-b25c-eaf316459f5e-node-exporter-textfile\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.723016 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.722862    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3907412a-9084-46b6-b25c-eaf316459f5e-node-exporter-accelerators-collector-config\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.723016 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.722910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc5cd\" (UniqueName: \"kubernetes.io/projected/94e94917-7911-4a16-a229-367716e81a26-kube-api-access-wc5cd\") pod \"openshift-state-metrics-b8f49f5c6-6mt7w\" (UID: \"94e94917-7911-4a16-a229-367716e81a26\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w"
Jul 29 22:51:31.723016 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.722950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/94e94917-7911-4a16-a229-367716e81a26-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-6mt7w\" (UID: \"94e94917-7911-4a16-a229-367716e81a26\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w"
Jul 29 22:51:31.723016 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.722978    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m4zh\" (UniqueName: \"kubernetes.io/projected/3907412a-9084-46b6-b25c-eaf316459f5e-kube-api-access-5m4zh\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.723210 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.723021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3907412a-9084-46b6-b25c-eaf316459f5e-root\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.723210 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.723050    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3907412a-9084-46b6-b25c-eaf316459f5e-node-exporter-tls\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.723210 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.723074    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3907412a-9084-46b6-b25c-eaf316459f5e-metrics-client-ca\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.723387 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.723311    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94e94917-7911-4a16-a229-367716e81a26-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-6mt7w\" (UID: \"94e94917-7911-4a16-a229-367716e81a26\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w"
Jul 29 22:51:31.726688 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.726657    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/94e94917-7911-4a16-a229-367716e81a26-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-6mt7w\" (UID: \"94e94917-7911-4a16-a229-367716e81a26\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w"
Jul 29 22:51:31.726837 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.726767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94e94917-7911-4a16-a229-367716e81a26-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-6mt7w\" (UID: \"94e94917-7911-4a16-a229-367716e81a26\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w"
Jul 29 22:51:31.733261 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.733238    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc5cd\" (UniqueName: \"kubernetes.io/projected/94e94917-7911-4a16-a229-367716e81a26-kube-api-access-wc5cd\") pod \"openshift-state-metrics-b8f49f5c6-6mt7w\" (UID: \"94e94917-7911-4a16-a229-367716e81a26\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w"
Jul 29 22:51:31.762666 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.762620    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:31.824199 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.824172    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3907412a-9084-46b6-b25c-eaf316459f5e-node-exporter-accelerators-collector-config\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.824611 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.824243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m4zh\" (UniqueName: \"kubernetes.io/projected/3907412a-9084-46b6-b25c-eaf316459f5e-kube-api-access-5m4zh\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.824611 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.824295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3907412a-9084-46b6-b25c-eaf316459f5e-root\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.824611 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.824321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3907412a-9084-46b6-b25c-eaf316459f5e-node-exporter-tls\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.824611 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.824345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3907412a-9084-46b6-b25c-eaf316459f5e-metrics-client-ca\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.824611 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.824378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3907412a-9084-46b6-b25c-eaf316459f5e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.824611 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.824427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3907412a-9084-46b6-b25c-eaf316459f5e-sys\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.824611 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.824450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3907412a-9084-46b6-b25c-eaf316459f5e-node-exporter-wtmp\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.824611 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.824475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3907412a-9084-46b6-b25c-eaf316459f5e-node-exporter-textfile\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.825020 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.824818    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3907412a-9084-46b6-b25c-eaf316459f5e-node-exporter-accelerators-collector-config\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.825020 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.824844    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3907412a-9084-46b6-b25c-eaf316459f5e-root\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.825020 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.824895    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3907412a-9084-46b6-b25c-eaf316459f5e-sys\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.825020 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.824923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3907412a-9084-46b6-b25c-eaf316459f5e-node-exporter-wtmp\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.825020 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.824972    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3907412a-9084-46b6-b25c-eaf316459f5e-node-exporter-textfile\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.825312 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.825289    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3907412a-9084-46b6-b25c-eaf316459f5e-metrics-client-ca\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.827349 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.827322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3907412a-9084-46b6-b25c-eaf316459f5e-node-exporter-tls\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.827450 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.827376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3907412a-9084-46b6-b25c-eaf316459f5e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.843967 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.838228    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m4zh\" (UniqueName: \"kubernetes.io/projected/3907412a-9084-46b6-b25c-eaf316459f5e-kube-api-access-5m4zh\") pod \"node-exporter-jrx9p\" (UID: \"3907412a-9084-46b6-b25c-eaf316459f5e\") " pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.846094 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.846073    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w"
Jul 29 22:51:31.862721 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.862654    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:31.907241 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.907189    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jrx9p"
Jul 29 22:51:31.967946 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:31.967784    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:32.063720 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.063673    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:32.166265 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.166181    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:32.264549 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.264517    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:32.363736 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.363701    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:32.384650 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.384623    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 22:51:32.399835 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.399791    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.403030 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.402948    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 22:51:32.403594 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.403520    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 29 22:51:32.403811 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.403776    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vcndz\""
Jul 29 22:51:32.403915 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.403778    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 29 22:51:32.404077 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.404054    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 29 22:51:32.404194 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.404100    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 29 22:51:32.404194 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.404107    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 29 22:51:32.404194 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.404054    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 29 22:51:32.404359 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.404326    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 29 22:51:32.406566 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.406400    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 29 22:51:32.406746 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.406725    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 29 22:51:32.437119 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.437058    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:32.530314 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.530269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.530314 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.530312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.530520 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.530339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-config-out\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.530520 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.530396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.530520 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.530447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.530520 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.530479    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.530520 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.530509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.530738 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.530598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.530738 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.530640    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-config-volume\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.530738 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.530675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnzcn\" (UniqueName: \"kubernetes.io/projected/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-kube-api-access-wnzcn\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.530738 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.530726    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-web-config\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.530909 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.530777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.530909 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.530818    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.632261 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.632230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.632447 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.632268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.632447 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.632317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.632447 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.632360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.632447 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.632388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-config-volume\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.632447 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.632419    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnzcn\" (UniqueName: \"kubernetes.io/projected/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-kube-api-access-wnzcn\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.632706 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.632454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-web-config\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.632706 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:51:32.632490    2581 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 29 22:51:32.632706 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.632513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.632706 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.632535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.632706 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:51:32.632561    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-main-tls podName:6ffa1afb-7773-4b55-b5d3-e46864abf5ff nodeName:}" failed. No retries permitted until 2026-07-29 22:51:33.132543108 +0000 UTC m=+315.690887905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "6ffa1afb-7773-4b55-b5d3-e46864abf5ff") : secret "alertmanager-main-tls" not found
Jul 29 22:51:32.632706 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.632646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.632706 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.632676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.632706 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.632707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-config-out\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.633112 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.632731    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.633290 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.633268    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.633780 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.633499    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.634063 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.634038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.636141 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.636093    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-web-config\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.636727 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.636705    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.636947 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.636906    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.637066 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.637044    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-config-out\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.637168 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.637126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.637384 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.637364    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-config-volume\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.637473 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.637447    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.638930 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.638904    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.645541 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.645524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnzcn\" (UniqueName: \"kubernetes.io/projected/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-kube-api-access-wnzcn\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:32.779583 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.779498    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:32.865626 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.865592    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:32.947268 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:51:32.947229    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3907412a_9084_46b6_b25c_eaf316459f5e.slice/crio-140b1c9ffc55a0ee085fe46c0ae272367c448ff771d28434c2c5af3c96f859cc WatchSource:0}: Error finding container 140b1c9ffc55a0ee085fe46c0ae272367c448ff771d28434c2c5af3c96f859cc: Status 404 returned error can't find the container with id 140b1c9ffc55a0ee085fe46c0ae272367c448ff771d28434c2c5af3c96f859cc
Jul 29 22:51:32.966263 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:32.966237    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:33.068375 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:33.068357    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w"]
Jul 29 22:51:33.070462 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:51:33.070436    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e94917_7911_4a16_a229_367716e81a26.slice/crio-d48144a23499796c6eb44340875f6b9065315bf8b9e5d6055cb0f9a5c86dee8b WatchSource:0}: Error finding container d48144a23499796c6eb44340875f6b9065315bf8b9e5d6055cb0f9a5c86dee8b: Status 404 returned error can't find the container with id d48144a23499796c6eb44340875f6b9065315bf8b9e5d6055cb0f9a5c86dee8b
Jul 29 22:51:33.136272 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:33.136248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:33.138604 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:33.138586    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:33.316505 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:33.316432    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:51:33.424859 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:33.424825    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:33.449409 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:33.449308    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 22:51:33.451939 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:51:33.451913    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ffa1afb_7773_4b55_b5d3_e46864abf5ff.slice/crio-779a07af105b5f121c9203d69bdfdb7869dfca4bcb3fea6a8cd34aafad2d6960 WatchSource:0}: Error finding container 779a07af105b5f121c9203d69bdfdb7869dfca4bcb3fea6a8cd34aafad2d6960: Status 404 returned error can't find the container with id 779a07af105b5f121c9203d69bdfdb7869dfca4bcb3fea6a8cd34aafad2d6960
Jul 29 22:51:33.509569 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:33.509541    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:33.870852 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:33.870789    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rr5g7" event={"ID":"2a191a01-962a-4268-93fc-639ea776d1b6","Type":"ContainerStarted","Data":"2a091987e0620491eb2db5f988aaa21a60dd1d7c709fcbe9355afd3e9f73d3d7"}
Jul 29 22:51:33.872276 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:33.872248    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ffa1afb-7773-4b55-b5d3-e46864abf5ff","Type":"ContainerStarted","Data":"779a07af105b5f121c9203d69bdfdb7869dfca4bcb3fea6a8cd34aafad2d6960"}
Jul 29 22:51:33.874504 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:33.874480    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w" event={"ID":"94e94917-7911-4a16-a229-367716e81a26","Type":"ContainerStarted","Data":"4602cde7bed257f5651247bf458b575ec0f16d1bd298dc1b1c61c78c8ffe3c5f"}
Jul 29 22:51:33.874704 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:33.874681    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w" event={"ID":"94e94917-7911-4a16-a229-367716e81a26","Type":"ContainerStarted","Data":"b798a7f81d001bc0629bce023f753e81d4ac0ac64da7bdb9af0a04e704aba021"}
Jul 29 22:51:33.874845 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:33.874735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w" event={"ID":"94e94917-7911-4a16-a229-367716e81a26","Type":"ContainerStarted","Data":"d48144a23499796c6eb44340875f6b9065315bf8b9e5d6055cb0f9a5c86dee8b"}
Jul 29 22:51:33.875772 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:33.875746    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jrx9p" event={"ID":"3907412a-9084-46b6-b25c-eaf316459f5e","Type":"ContainerStarted","Data":"140b1c9ffc55a0ee085fe46c0ae272367c448ff771d28434c2c5af3c96f859cc"}
Jul 29 22:51:33.889654 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:33.889609    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rr5g7" podStartSLOduration=33.704948033 podStartE2EDuration="45.889591863s" totalImagesPullingTime="12.18464383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:50:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:33.88872163 +0000 UTC m=+316.447066445" watchObservedRunningTime="2026-07-29 22:51:33.889591863 +0000 UTC m=+316.447936680"
Jul 29 22:51:34.308678 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:34.308520    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:34.415664 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:34.415626    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:34.519240 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:34.519109    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:34.706543 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:34.706520    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:34.808512 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:34.808452    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:34.881096 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:34.881066    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerID="0213eda7c803c6ccac30b8ad74e0fca65e27d083cd95c2a93ac14710a8ddd123" exitCode=0
Jul 29 22:51:34.881402 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:34.881145    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ffa1afb-7773-4b55-b5d3-e46864abf5ff","Type":"ContainerDied","Data":"0213eda7c803c6ccac30b8ad74e0fca65e27d083cd95c2a93ac14710a8ddd123"}
Jul 29 22:51:34.883409 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:34.883388    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w" event={"ID":"94e94917-7911-4a16-a229-367716e81a26","Type":"ContainerStarted","Data":"f6da6dfa1fb8d179285a76457ca8c901f24a7d47865166ea0e42bcbd526beacd"}
Jul 29 22:51:34.884944 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:34.884922    2581 generic.go:356] "Generic (PLEG): container finished" podID="3907412a-9084-46b6-b25c-eaf316459f5e" containerID="21e6db602374782aed2e7a0bd9fa70b03f4af3eeeff3a03a51f5d27ffbf3239a" exitCode=0
Jul 29 22:51:34.885036 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:34.884994    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jrx9p" event={"ID":"3907412a-9084-46b6-b25c-eaf316459f5e","Type":"ContainerDied","Data":"21e6db602374782aed2e7a0bd9fa70b03f4af3eeeff3a03a51f5d27ffbf3239a"}
Jul 29 22:51:34.908339 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:34.908315    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:34.953506 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:34.953466    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6mt7w" podStartSLOduration=2.848758974 podStartE2EDuration="3.953451956s" totalImagesPullingTime="1.104692982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:34.951892444 +0000 UTC m=+317.510237260" watchObservedRunningTime="2026-07-29 22:51:34.953451956 +0000 UTC m=+317.511796770"
Jul 29 22:51:35.392578 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:35.392550    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c676dc74-4k7lf"]
Jul 29 22:51:35.476988 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:35.476952    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:35.577995 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:35.577963    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:35.892447 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:35.892409    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jrx9p" event={"ID":"3907412a-9084-46b6-b25c-eaf316459f5e","Type":"ContainerStarted","Data":"da2a81d0000cd04aa6a979952c053e86774131646bb8cb2e363404cf58271f8d"}
Jul 29 22:51:35.892887 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:35.892481    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jrx9p" event={"ID":"3907412a-9084-46b6-b25c-eaf316459f5e","Type":"ContainerStarted","Data":"445072b6171e350a633cad07e5ab11146d5eeaef39b44b8de5ef2060d2611d6a"}
Jul 29 22:51:35.916949 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:35.916899    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jrx9p" podStartSLOduration=3.878576247 podStartE2EDuration="4.916878563s" totalImagesPullingTime="966.709122ms" totalInitContainerRuntime="71.593194ms" isStatefulPod=true podCreationTimestamp="2026-07-29 22:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:35.913889932 +0000 UTC m=+318.472234749" watchObservedRunningTime="2026-07-29 22:51:35.916878563 +0000 UTC m=+318.475223379"
Jul 29 22:51:36.069956 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:36.069927    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:36.167007 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:36.166922    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:36.266738 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:36.266707    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:36.901007 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:36.900970    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ffa1afb-7773-4b55-b5d3-e46864abf5ff","Type":"ContainerStarted","Data":"cc6e834a11965a340a370b0000ecd732835b1ab2ab23486b8b744f5d173c3178"}
Jul 29 22:51:36.901318 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:36.901066    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ffa1afb-7773-4b55-b5d3-e46864abf5ff","Type":"ContainerStarted","Data":"ebb99a544e5efe2ba5aae35d5f786ad982d3f4ea73e655dcae06be1f6564ed05"}
Jul 29 22:51:36.901318 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:36.901097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ffa1afb-7773-4b55-b5d3-e46864abf5ff","Type":"ContainerStarted","Data":"387642b8a776d5e4b8c2b478a91bfc5275846447e511263b70bc225207de6838"}
Jul 29 22:51:36.901318 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:36.901122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ffa1afb-7773-4b55-b5d3-e46864abf5ff","Type":"ContainerStarted","Data":"1ba7b1b023ce7ccac5a7376d2c7ee342a37ba6ba25a23e370299e40367fa9cf4"}
Jul 29 22:51:36.901318 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:36.901146    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ffa1afb-7773-4b55-b5d3-e46864abf5ff","Type":"ContainerStarted","Data":"c1bf3521ed0e6dec36e55ab3bf3513ac3baf01bda17baf066e8f75d60f47cf17"}
Jul 29 22:51:37.548598 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.548543    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:37.549295 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.549270    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 22:51:37.553204 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.553182    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.556024 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.555997    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 29 22:51:37.556254 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.556234    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 29 22:51:37.556334 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.556310    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 29 22:51:37.556885 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.556853    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 29 22:51:37.556993 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.556917    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 29 22:51:37.557550 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.557149    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 29 22:51:37.557550 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.557216    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 29 22:51:37.557550 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.557289    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 29 22:51:37.557550 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.557290    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 29 22:51:37.557550 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.557214    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 29 22:51:37.557550 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.557416    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 29 22:51:37.561061 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.558730    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7dl2bq2qmt4bd\""
Jul 29 22:51:37.561061 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.558989    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4qh47\""
Jul 29 22:51:37.561061 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.559225    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 29 22:51:37.561061 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.559425    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 29 22:51:37.561061 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.559636    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 29 22:51:37.562388 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.562366    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 29 22:51:37.570274 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.570253    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 22:51:37.676422 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.676386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84e9f303-8aa4-4610-be99-1c8ce985cf7c-config-out\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.676422 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.676420    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.676641 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.676464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.676641 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.676499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-web-config\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.676641 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.676516    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.676641 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.676572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.676641 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.676631    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfgcc\" (UniqueName: \"kubernetes.io/projected/84e9f303-8aa4-4610-be99-1c8ce985cf7c-kube-api-access-qfgcc\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.676927 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.676677    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.676927 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.676701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.676927 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.676742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-config\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.676927 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.676764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84e9f303-8aa4-4610-be99-1c8ce985cf7c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.676927 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.676824    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.676927 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.676872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.676927 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.676921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.677272 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.676956    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.677272 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.677016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.677272 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.677069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.677272 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.677264    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.677467 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.677294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.677467 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.677312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.778821 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.778772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-config\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.778949 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.778828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84e9f303-8aa4-4610-be99-1c8ce985cf7c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.778949 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.778861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.778949 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.778887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.778949 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.778911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.778949 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.778942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.779208 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.778995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.779208 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.779022    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.779208 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.779049    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.779208 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.779067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.779208 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.779085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.779208 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.779117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84e9f303-8aa4-4610-be99-1c8ce985cf7c-config-out\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.779208 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.779131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.779208 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.779149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.779208 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.779168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-web-config\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.779208 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.779183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.779208 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.779198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.779727 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.779223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfgcc\" (UniqueName: \"kubernetes.io/projected/84e9f303-8aa4-4610-be99-1c8ce985cf7c-kube-api-access-qfgcc\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.779727 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.779276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.779727 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.779305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.780697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.780486    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.781269 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.781241    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.781743 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.781715    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.782021 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.781995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.782114 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.782067    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.782546 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.782506    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.783187 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.783074    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-web-config\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.783187 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.783116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-config\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.783336 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.783208    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.783336 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.783289    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.783451 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.783415    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.783925 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.783905    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84e9f303-8aa4-4610-be99-1c8ce985cf7c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.784075 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.784051    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.784321 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.784267    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.784737 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.784710    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84e9f303-8aa4-4610-be99-1c8ce985cf7c-config-out\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.785954 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.785928    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.786036 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.785960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.788107 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.786488    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.788561 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.788527    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.790994 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.790970    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfgcc\" (UniqueName: \"kubernetes.io/projected/84e9f303-8aa4-4610-be99-1c8ce985cf7c-kube-api-access-qfgcc\") pod \"prometheus-k8s-0\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.868209 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.868084    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:37.910706 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.910675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ffa1afb-7773-4b55-b5d3-e46864abf5ff","Type":"ContainerStarted","Data":"12cbad704322c61f7af365e384e7987e02b36bccb522f1280243640e2ca5acb9"}
Jul 29 22:51:37.936131 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:37.935937    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.154418434 podStartE2EDuration="5.935916856s" totalImagesPullingTime="3.763588637s" totalInitContainerRuntime="17.909785ms" isStatefulPod=false podCreationTimestamp="2026-07-29 22:51:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:37.934258178 +0000 UTC m=+320.492603221" watchObservedRunningTime="2026-07-29 22:51:37.935916856 +0000 UTC m=+320.494261673"
Jul 29 22:51:38.026614 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:38.026583    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 22:51:38.029124 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:51:38.029094    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e9f303_8aa4_4610_be99_1c8ce985cf7c.slice/crio-a97e82b4139230a2eb2156e86f6b0d81ddfaff775fa73f748fe52c16d682c849 WatchSource:0}: Error finding container a97e82b4139230a2eb2156e86f6b0d81ddfaff775fa73f748fe52c16d682c849: Status 404 returned error can't find the container with id a97e82b4139230a2eb2156e86f6b0d81ddfaff775fa73f748fe52c16d682c849
Jul 29 22:51:38.254009 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:38.253976    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:38.319501 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:38.319472    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:38.916110 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:38.916078    2581 generic.go:356] "Generic (PLEG): container finished" podID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerID="6603d8ebc8461c8276c43a7323c31aafeddd1a46879105b07d9f4be68cd9b855" exitCode=0
Jul 29 22:51:38.916490 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:38.916161    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84e9f303-8aa4-4610-be99-1c8ce985cf7c","Type":"ContainerDied","Data":"6603d8ebc8461c8276c43a7323c31aafeddd1a46879105b07d9f4be68cd9b855"}
Jul 29 22:51:38.916490 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:38.916246    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84e9f303-8aa4-4610-be99-1c8ce985cf7c","Type":"ContainerStarted","Data":"a97e82b4139230a2eb2156e86f6b0d81ddfaff775fa73f748fe52c16d682c849"}
Jul 29 22:51:40.438865 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:40.438816    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:40.539028 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:40.538997    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:40.639465 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:40.639427    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:40.695858 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:40.695765    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:41.908300 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:41.908260    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:42.004534 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:42.004498    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:42.935208 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:42.935173    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84e9f303-8aa4-4610-be99-1c8ce985cf7c","Type":"ContainerStarted","Data":"6012c5348b1151b23e9b0f98584e3a219c5b2d1aeb923960f080a8cf8a5a0ab6"}
Jul 29 22:51:42.935712 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:42.935272    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84e9f303-8aa4-4610-be99-1c8ce985cf7c","Type":"ContainerStarted","Data":"9c37354c6f76e060e30744afd27f2ba020170e5c4ab9929697280b063b2af572"}
Jul 29 22:51:44.626522 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:44.626488    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:44.679025 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:44.678997    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:44.946877 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:44.946780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84e9f303-8aa4-4610-be99-1c8ce985cf7c","Type":"ContainerStarted","Data":"b9eb7c26626169f0e224062af0252b8dd86bebb447ab48814a9f2d8224d0d239"}
Jul 29 22:51:44.946877 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:44.946862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84e9f303-8aa4-4610-be99-1c8ce985cf7c","Type":"ContainerStarted","Data":"e975fa7ff4d6bf7d2b33d812a845714435b953840401d15a3609bf10f015db0d"}
Jul 29 22:51:44.947025 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:44.946890    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84e9f303-8aa4-4610-be99-1c8ce985cf7c","Type":"ContainerStarted","Data":"663bfa3e16bc9ea350a0c823979217c5369897173096c1995fd5c98398f3a6af"}
Jul 29 22:51:44.947025 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:44.946921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84e9f303-8aa4-4610-be99-1c8ce985cf7c","Type":"ContainerStarted","Data":"d1fb2e40c4e562f882abb1140720382fe82175951d06df8939362f4e8aef2ef4"}
Jul 29 22:51:44.977245 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:44.977197    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.960417829 podStartE2EDuration="7.977182607s" totalImagesPullingTime="4.996135066s" totalInitContainerRuntime="20.629712ms" isStatefulPod=false podCreationTimestamp="2026-07-29 22:51:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:44.975297966 +0000 UTC m=+327.533642783" watchObservedRunningTime="2026-07-29 22:51:44.977182607 +0000 UTC m=+327.535527423"
Jul 29 22:51:47.869076 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:47.869040    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:51:48.112639 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:48.112596    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:48.214111 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:48.214050    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:48.827429 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:48.827391    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:54.485887 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:54.485850    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6ffa1afb-7773-4b55-b5d3-e46864abf5ff/init-config-reloader/0.log"
Jul 29 22:51:54.493573 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:54.493551    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6ffa1afb-7773-4b55-b5d3-e46864abf5ff/alertmanager/0.log"
Jul 29 22:51:54.563162 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:54.563135    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6ffa1afb-7773-4b55-b5d3-e46864abf5ff/config-reloader/0.log"
Jul 29 22:51:54.763331 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:54.763253    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6ffa1afb-7773-4b55-b5d3-e46864abf5ff/kube-rbac-proxy-web/0.log"
Jul 29 22:51:54.963886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:54.963860    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6ffa1afb-7773-4b55-b5d3-e46864abf5ff/kube-rbac-proxy/0.log"
Jul 29 22:51:55.163329 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:55.163297    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6ffa1afb-7773-4b55-b5d3-e46864abf5ff/kube-rbac-proxy-metric/0.log"
Jul 29 22:51:55.362931 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:55.362905    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6ffa1afb-7773-4b55-b5d3-e46864abf5ff/prom-label-proxy/0.log"
Jul 29 22:51:55.621278 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:55.621240    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:55.720401 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:55.720372    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:57.362829 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:57.362787    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jrx9p_3907412a-9084-46b6-b25c-eaf316459f5e/init-textfile/0.log"
Jul 29 22:51:57.563951 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:57.563918    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jrx9p_3907412a-9084-46b6-b25c-eaf316459f5e/node-exporter/0.log"
Jul 29 22:51:57.762753 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:57.762690    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jrx9p_3907412a-9084-46b6-b25c-eaf316459f5e/kube-rbac-proxy/0.log"
Jul 29 22:51:58.272813 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:58.272777    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:58.336956 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:58.336916    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:51:58.562820 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:58.562728    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-6mt7w_94e94917-7911-4a16-a229-367716e81a26/kube-rbac-proxy-main/0.log"
Jul 29 22:51:58.763943 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:58.763916    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-6mt7w_94e94917-7911-4a16-a229-367716e81a26/kube-rbac-proxy-self/0.log"
Jul 29 22:51:58.963725 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:58.963700    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-6mt7w_94e94917-7911-4a16-a229-367716e81a26/openshift-state-metrics/0.log"
Jul 29 22:51:59.162781 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:59.162755    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_84e9f303-8aa4-4610-be99-1c8ce985cf7c/init-config-reloader/0.log"
Jul 29 22:51:59.364416 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:59.364386    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_84e9f303-8aa4-4610-be99-1c8ce985cf7c/prometheus/0.log"
Jul 29 22:51:59.563937 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:59.563913    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_84e9f303-8aa4-4610-be99-1c8ce985cf7c/config-reloader/0.log"
Jul 29 22:51:59.763688 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:59.763622    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_84e9f303-8aa4-4610-be99-1c8ce985cf7c/thanos-sidecar/0.log"
Jul 29 22:51:59.963125 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:51:59.963106    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_84e9f303-8aa4-4610-be99-1c8ce985cf7c/kube-rbac-proxy-web/0.log"
Jul 29 22:52:00.162605 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.162577    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_84e9f303-8aa4-4610-be99-1c8ce985cf7c/kube-rbac-proxy/0.log"
Jul 29 22:52:00.362885 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.362858    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_84e9f303-8aa4-4610-be99-1c8ce985cf7c/kube-rbac-proxy-thanos/0.log"
Jul 29 22:52:00.410950 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.410769    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5c676dc74-4k7lf" podUID="d7e081dc-a705-4255-aed8-c20f7090c5b3" containerName="registry" containerID="cri-o://a769bb6e937a23456737b247e48ba37eba3d6885ac8ce19e84b14b1d52764bef" gracePeriod=30
Jul 29 22:52:00.567025 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.566953    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-vq8rs_7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8/prometheus-operator/0.log"
Jul 29 22:52:00.691974 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.691920    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:52:00.762505 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.762486    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-vq8rs_7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8/kube-rbac-proxy/0.log"
Jul 29 22:52:00.767925 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.767907    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7e081dc-a705-4255-aed8-c20f7090c5b3-image-registry-private-configuration\") pod \"d7e081dc-a705-4255-aed8-c20f7090c5b3\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") "
Jul 29 22:52:00.767992 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.767937    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-bound-sa-token\") pod \"d7e081dc-a705-4255-aed8-c20f7090c5b3\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") "
Jul 29 22:52:00.768144 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.768128    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7e081dc-a705-4255-aed8-c20f7090c5b3-installation-pull-secrets\") pod \"d7e081dc-a705-4255-aed8-c20f7090c5b3\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") "
Jul 29 22:52:00.768201 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.768162    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7e081dc-a705-4255-aed8-c20f7090c5b3-ca-trust-extracted\") pod \"d7e081dc-a705-4255-aed8-c20f7090c5b3\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") "
Jul 29 22:52:00.768258 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.768242    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxgsb\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-kube-api-access-zxgsb\") pod \"d7e081dc-a705-4255-aed8-c20f7090c5b3\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") "
Jul 29 22:52:00.768307 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.768275    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-certificates\") pod \"d7e081dc-a705-4255-aed8-c20f7090c5b3\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") "
Jul 29 22:52:00.768342 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.768305    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls\") pod \"d7e081dc-a705-4255-aed8-c20f7090c5b3\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") "
Jul 29 22:52:00.768391 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.768337    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7e081dc-a705-4255-aed8-c20f7090c5b3-trusted-ca\") pod \"d7e081dc-a705-4255-aed8-c20f7090c5b3\" (UID: \"d7e081dc-a705-4255-aed8-c20f7090c5b3\") "
Jul 29 22:52:00.768871 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.768722    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d7e081dc-a705-4255-aed8-c20f7090c5b3" (UID: "d7e081dc-a705-4255-aed8-c20f7090c5b3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:52:00.769017 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.768982    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e081dc-a705-4255-aed8-c20f7090c5b3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d7e081dc-a705-4255-aed8-c20f7090c5b3" (UID: "d7e081dc-a705-4255-aed8-c20f7090c5b3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:52:00.770998 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.770968    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e081dc-a705-4255-aed8-c20f7090c5b3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d7e081dc-a705-4255-aed8-c20f7090c5b3" (UID: "d7e081dc-a705-4255-aed8-c20f7090c5b3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:00.771098 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.771010    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d7e081dc-a705-4255-aed8-c20f7090c5b3" (UID: "d7e081dc-a705-4255-aed8-c20f7090c5b3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:52:00.771098 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.771038    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d7e081dc-a705-4255-aed8-c20f7090c5b3" (UID: "d7e081dc-a705-4255-aed8-c20f7090c5b3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:52:00.771098 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.771050    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e081dc-a705-4255-aed8-c20f7090c5b3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d7e081dc-a705-4255-aed8-c20f7090c5b3" (UID: "d7e081dc-a705-4255-aed8-c20f7090c5b3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:00.771265 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.771110    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-kube-api-access-zxgsb" (OuterVolumeSpecName: "kube-api-access-zxgsb") pod "d7e081dc-a705-4255-aed8-c20f7090c5b3" (UID: "d7e081dc-a705-4255-aed8-c20f7090c5b3"). InnerVolumeSpecName "kube-api-access-zxgsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:52:00.776713 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.776692    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e081dc-a705-4255-aed8-c20f7090c5b3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d7e081dc-a705-4255-aed8-c20f7090c5b3" (UID: "d7e081dc-a705-4255-aed8-c20f7090c5b3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:52:00.869701 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.869680    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxgsb\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-kube-api-access-zxgsb\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:00.869701 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.869699    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-certificates\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:00.869897 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.869709    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-registry-tls\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:00.869897 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.869718    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7e081dc-a705-4255-aed8-c20f7090c5b3-trusted-ca\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:00.869897 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.869726    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7e081dc-a705-4255-aed8-c20f7090c5b3-image-registry-private-configuration\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:00.869897 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.869734    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7e081dc-a705-4255-aed8-c20f7090c5b3-bound-sa-token\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:00.869897 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.869743    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7e081dc-a705-4255-aed8-c20f7090c5b3-installation-pull-secrets\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:00.869897 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.869750    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7e081dc-a705-4255-aed8-c20f7090c5b3-ca-trust-extracted\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:00.963411 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:00.963360    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-98g9k_0cbd4a90-8b07-483b-9fc6-e3ac61d55df9/prometheus-operator-admission-webhook/0.log"
Jul 29 22:52:01.006469 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:01.006443    2581 generic.go:356] "Generic (PLEG): container finished" podID="d7e081dc-a705-4255-aed8-c20f7090c5b3" containerID="a769bb6e937a23456737b247e48ba37eba3d6885ac8ce19e84b14b1d52764bef" exitCode=0
Jul 29 22:52:01.006558 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:01.006489    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c676dc74-4k7lf"
Jul 29 22:52:01.006558 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:01.006526    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c676dc74-4k7lf" event={"ID":"d7e081dc-a705-4255-aed8-c20f7090c5b3","Type":"ContainerDied","Data":"a769bb6e937a23456737b247e48ba37eba3d6885ac8ce19e84b14b1d52764bef"}
Jul 29 22:52:01.006630 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:01.006568    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c676dc74-4k7lf" event={"ID":"d7e081dc-a705-4255-aed8-c20f7090c5b3","Type":"ContainerDied","Data":"849a5d1bab6a6f9b7201c62f3c0d4e0b75000b3e202f08f7f32818c49c297e87"}
Jul 29 22:52:01.006630 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:01.006590    2581 scope.go:117] "RemoveContainer" containerID="a769bb6e937a23456737b247e48ba37eba3d6885ac8ce19e84b14b1d52764bef"
Jul 29 22:52:01.015216 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:01.015199    2581 scope.go:117] "RemoveContainer" containerID="a769bb6e937a23456737b247e48ba37eba3d6885ac8ce19e84b14b1d52764bef"
Jul 29 22:52:01.015455 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:52:01.015438    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a769bb6e937a23456737b247e48ba37eba3d6885ac8ce19e84b14b1d52764bef\": container with ID starting with a769bb6e937a23456737b247e48ba37eba3d6885ac8ce19e84b14b1d52764bef not found: ID does not exist" containerID="a769bb6e937a23456737b247e48ba37eba3d6885ac8ce19e84b14b1d52764bef"
Jul 29 22:52:01.015516 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:01.015460    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a769bb6e937a23456737b247e48ba37eba3d6885ac8ce19e84b14b1d52764bef"} err="failed to get container status \"a769bb6e937a23456737b247e48ba37eba3d6885ac8ce19e84b14b1d52764bef\": rpc error: code = NotFound desc = could not find container \"a769bb6e937a23456737b247e48ba37eba3d6885ac8ce19e84b14b1d52764bef\": container with ID starting with a769bb6e937a23456737b247e48ba37eba3d6885ac8ce19e84b14b1d52764bef not found: ID does not exist"
Jul 29 22:52:01.027256 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:01.027235    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c676dc74-4k7lf"]
Jul 29 22:52:01.031329 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:01.031310    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5c676dc74-4k7lf"]
Jul 29 22:52:01.951356 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:01.951323    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7e081dc-a705-4255-aed8-c20f7090c5b3" path="/var/lib/kubelet/pods/d7e081dc-a705-4255-aed8-c20f7090c5b3/volumes"
Jul 29 22:52:04.233425 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:04.233381    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:52:04.290075 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:04.290048    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:52:04.658512 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:04.658477    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:52:04.759327 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:04.759274    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:52:08.038500 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:08.038462    2581 generic.go:356] "Generic (PLEG): container finished" podID="b6921abb-a2c9-4b8a-9cf5-6c0d694c2630" containerID="d930b9224b768b2d009fc9f9deebdeb442a024275b35c9fd392b4301eb91ec4a" exitCode=0
Jul 29 22:52:08.038919 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:08.038506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq" event={"ID":"b6921abb-a2c9-4b8a-9cf5-6c0d694c2630","Type":"ContainerDied","Data":"d930b9224b768b2d009fc9f9deebdeb442a024275b35c9fd392b4301eb91ec4a"}
Jul 29 22:52:08.038975 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:08.038927    2581 scope.go:117] "RemoveContainer" containerID="d930b9224b768b2d009fc9f9deebdeb442a024275b35c9fd392b4301eb91ec4a"
Jul 29 22:52:09.044685 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:09.044652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq" event={"ID":"b6921abb-a2c9-4b8a-9cf5-6c0d694c2630","Type":"ContainerStarted","Data":"a8457ef0252dc3b3752d8ff6fee952cc8305a302dc9468428c010a7194c5e58a"}
Jul 29 22:52:10.619249 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:10.619161    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:52:10.676062 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:10.676034    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:52:31.826223 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:31.826194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs\") pod \"network-metrics-daemon-2ksj4\" (UID: \"cf450310-9f32-4f65-96ea-b4d676ff2df7\") " pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:52:31.828444 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:31.828421    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf450310-9f32-4f65-96ea-b4d676ff2df7-metrics-certs\") pod \"network-metrics-daemon-2ksj4\" (UID: \"cf450310-9f32-4f65-96ea-b4d676ff2df7\") " pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:52:31.851055 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:31.851033    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wkbjs\""
Jul 29 22:52:31.858211 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:31.858197    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ksj4"
Jul 29 22:52:31.981569 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:31.981438    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2ksj4"]
Jul 29 22:52:31.984339 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:52:31.984313    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf450310_9f32_4f65_96ea_b4d676ff2df7.slice/crio-d2d4e6e0c3ded9bbfaa2f0042bfb69810feb60b19ee3d555606e7b29b732fe01 WatchSource:0}: Error finding container d2d4e6e0c3ded9bbfaa2f0042bfb69810feb60b19ee3d555606e7b29b732fe01: Status 404 returned error can't find the container with id d2d4e6e0c3ded9bbfaa2f0042bfb69810feb60b19ee3d555606e7b29b732fe01
Jul 29 22:52:32.128445 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:32.128409    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2ksj4" event={"ID":"cf450310-9f32-4f65-96ea-b4d676ff2df7","Type":"ContainerStarted","Data":"d2d4e6e0c3ded9bbfaa2f0042bfb69810feb60b19ee3d555606e7b29b732fe01"}
Jul 29 22:52:33.134257 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:33.134229    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2ksj4" event={"ID":"cf450310-9f32-4f65-96ea-b4d676ff2df7","Type":"ContainerStarted","Data":"4ddb828afa7ce9a0bd460838efd4a93c802117a71d6be7bd2a1944ec4f7cc783"}
Jul 29 22:52:33.134564 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:33.134280    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2ksj4" event={"ID":"cf450310-9f32-4f65-96ea-b4d676ff2df7","Type":"ContainerStarted","Data":"c26cacea360a502157ac1ba8ada2d37006921d2682e9d84d398b54e63f3638ef"}
Jul 29 22:52:33.152550 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:33.152507    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2ksj4" podStartSLOduration=374.222828334 podStartE2EDuration="6m15.152491405s" totalImagesPullingTime="929.663071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:52:33.149498489 +0000 UTC m=+375.707843306" watchObservedRunningTime="2026-07-29 22:52:33.152491405 +0000 UTC m=+375.710836221"
Jul 29 22:52:33.241825 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:33.241755    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:52:33.342463 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:33.342434    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:52:37.868615 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:37.868573    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:37.889895 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:37.889868    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:38.168180 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:38.168119    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:39.881020 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:39.880984    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:52:39.940332 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:39.940302    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:52:47.461320 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:47.461283    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:52:47.517345 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:47.517311    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:52:48.841333 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:48.841302    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:52:48.942534 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:48.942507    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:52:51.772396 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:51.772365    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 22:52:51.772779 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:51.772627    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="alertmanager" containerID="cri-o://c1bf3521ed0e6dec36e55ab3bf3513ac3baf01bda17baf066e8f75d60f47cf17" gracePeriod=120
Jul 29 22:52:51.772779 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:51.772716    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="kube-rbac-proxy-web" containerID="cri-o://387642b8a776d5e4b8c2b478a91bfc5275846447e511263b70bc225207de6838" gracePeriod=120
Jul 29 22:52:51.772779 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:51.772707    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="kube-rbac-proxy-metric" containerID="cri-o://cc6e834a11965a340a370b0000ecd732835b1ab2ab23486b8b744f5d173c3178" gracePeriod=120
Jul 29 22:52:51.772976 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:51.772782    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="kube-rbac-proxy" containerID="cri-o://ebb99a544e5efe2ba5aae35d5f786ad982d3f4ea73e655dcae06be1f6564ed05" gracePeriod=120
Jul 29 22:52:51.772976 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:51.772740    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="prom-label-proxy" containerID="cri-o://12cbad704322c61f7af365e384e7987e02b36bccb522f1280243640e2ca5acb9" gracePeriod=120
Jul 29 22:52:51.772976 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:51.772821    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="config-reloader" containerID="cri-o://1ba7b1b023ce7ccac5a7376d2c7ee342a37ba6ba25a23e370299e40367fa9cf4" gracePeriod=120
Jul 29 22:52:52.206327 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:52.206302    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerID="12cbad704322c61f7af365e384e7987e02b36bccb522f1280243640e2ca5acb9" exitCode=0
Jul 29 22:52:52.206327 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:52.206324    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerID="ebb99a544e5efe2ba5aae35d5f786ad982d3f4ea73e655dcae06be1f6564ed05" exitCode=0
Jul 29 22:52:52.206483 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:52.206333    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerID="1ba7b1b023ce7ccac5a7376d2c7ee342a37ba6ba25a23e370299e40367fa9cf4" exitCode=0
Jul 29 22:52:52.206483 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:52.206339    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerID="c1bf3521ed0e6dec36e55ab3bf3513ac3baf01bda17baf066e8f75d60f47cf17" exitCode=0
Jul 29 22:52:52.206483 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:52.206372    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ffa1afb-7773-4b55-b5d3-e46864abf5ff","Type":"ContainerDied","Data":"12cbad704322c61f7af365e384e7987e02b36bccb522f1280243640e2ca5acb9"}
Jul 29 22:52:52.206483 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:52.206435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ffa1afb-7773-4b55-b5d3-e46864abf5ff","Type":"ContainerDied","Data":"ebb99a544e5efe2ba5aae35d5f786ad982d3f4ea73e655dcae06be1f6564ed05"}
Jul 29 22:52:52.206483 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:52.206462    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ffa1afb-7773-4b55-b5d3-e46864abf5ff","Type":"ContainerDied","Data":"1ba7b1b023ce7ccac5a7376d2c7ee342a37ba6ba25a23e370299e40367fa9cf4"}
Jul 29 22:52:52.206483 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:52.206486    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ffa1afb-7773-4b55-b5d3-e46864abf5ff","Type":"ContainerDied","Data":"c1bf3521ed0e6dec36e55ab3bf3513ac3baf01bda17baf066e8f75d60f47cf17"}
Jul 29 22:52:53.012212 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.012190    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.189520 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.189489    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnzcn\" (UniqueName: \"kubernetes.io/projected/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-kube-api-access-wnzcn\") pod \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") "
Jul 29 22:52:53.189681 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.189531    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-kube-rbac-proxy\") pod \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") "
Jul 29 22:52:53.189681 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.189557    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-kube-rbac-proxy-web\") pod \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") "
Jul 29 22:52:53.189681 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.189582    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-alertmanager-trusted-ca-bundle\") pod \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") "
Jul 29 22:52:53.189681 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.189602    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-config-volume\") pod \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") "
Jul 29 22:52:53.189681 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.189619    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-metrics-client-ca\") pod \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") "
Jul 29 22:52:53.189681 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.189636    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-main-tls\") pod \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") "
Jul 29 22:52:53.189681 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.189663    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-config-out\") pod \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") "
Jul 29 22:52:53.190058 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.189689    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-tls-assets\") pod \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") "
Jul 29 22:52:53.190058 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.189717    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-web-config\") pod \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") "
Jul 29 22:52:53.190058 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.189763    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") "
Jul 29 22:52:53.190058 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.189829    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-alertmanager-main-db\") pod \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") "
Jul 29 22:52:53.190058 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.189888    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-cluster-tls-config\") pod \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\" (UID: \"6ffa1afb-7773-4b55-b5d3-e46864abf5ff\") "
Jul 29 22:52:53.190058 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.190042    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6ffa1afb-7773-4b55-b5d3-e46864abf5ff" (UID: "6ffa1afb-7773-4b55-b5d3-e46864abf5ff"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:52:53.190356 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.190163    2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:53.190356 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.190048    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6ffa1afb-7773-4b55-b5d3-e46864abf5ff" (UID: "6ffa1afb-7773-4b55-b5d3-e46864abf5ff"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:52:53.190947 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.190881    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6ffa1afb-7773-4b55-b5d3-e46864abf5ff" (UID: "6ffa1afb-7773-4b55-b5d3-e46864abf5ff"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:52:53.192548 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.192513    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6ffa1afb-7773-4b55-b5d3-e46864abf5ff" (UID: "6ffa1afb-7773-4b55-b5d3-e46864abf5ff"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:53.192646 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.192582    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-config-volume" (OuterVolumeSpecName: "config-volume") pod "6ffa1afb-7773-4b55-b5d3-e46864abf5ff" (UID: "6ffa1afb-7773-4b55-b5d3-e46864abf5ff"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:53.192646 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.192597    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6ffa1afb-7773-4b55-b5d3-e46864abf5ff" (UID: "6ffa1afb-7773-4b55-b5d3-e46864abf5ff"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:53.192948 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.192870    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6ffa1afb-7773-4b55-b5d3-e46864abf5ff" (UID: "6ffa1afb-7773-4b55-b5d3-e46864abf5ff"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:53.192948 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.192917    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-kube-api-access-wnzcn" (OuterVolumeSpecName: "kube-api-access-wnzcn") pod "6ffa1afb-7773-4b55-b5d3-e46864abf5ff" (UID: "6ffa1afb-7773-4b55-b5d3-e46864abf5ff"). InnerVolumeSpecName "kube-api-access-wnzcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:52:53.193091 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.193003    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6ffa1afb-7773-4b55-b5d3-e46864abf5ff" (UID: "6ffa1afb-7773-4b55-b5d3-e46864abf5ff"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:53.193852 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.193818    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6ffa1afb-7773-4b55-b5d3-e46864abf5ff" (UID: "6ffa1afb-7773-4b55-b5d3-e46864abf5ff"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:52:53.194036 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.194016    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-config-out" (OuterVolumeSpecName: "config-out") pod "6ffa1afb-7773-4b55-b5d3-e46864abf5ff" (UID: "6ffa1afb-7773-4b55-b5d3-e46864abf5ff"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:52:53.196562 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.196540    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6ffa1afb-7773-4b55-b5d3-e46864abf5ff" (UID: "6ffa1afb-7773-4b55-b5d3-e46864abf5ff"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:53.202908 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.202885    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-web-config" (OuterVolumeSpecName: "web-config") pod "6ffa1afb-7773-4b55-b5d3-e46864abf5ff" (UID: "6ffa1afb-7773-4b55-b5d3-e46864abf5ff"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:53.212958 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.212936    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerID="cc6e834a11965a340a370b0000ecd732835b1ab2ab23486b8b744f5d173c3178" exitCode=0
Jul 29 22:52:53.212958 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.212958    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerID="387642b8a776d5e4b8c2b478a91bfc5275846447e511263b70bc225207de6838" exitCode=0
Jul 29 22:52:53.213077 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.212987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ffa1afb-7773-4b55-b5d3-e46864abf5ff","Type":"ContainerDied","Data":"cc6e834a11965a340a370b0000ecd732835b1ab2ab23486b8b744f5d173c3178"}
Jul 29 22:52:53.213077 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.213018    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.213077 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.213066    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ffa1afb-7773-4b55-b5d3-e46864abf5ff","Type":"ContainerDied","Data":"387642b8a776d5e4b8c2b478a91bfc5275846447e511263b70bc225207de6838"}
Jul 29 22:52:53.213172 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.213094    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ffa1afb-7773-4b55-b5d3-e46864abf5ff","Type":"ContainerDied","Data":"779a07af105b5f121c9203d69bdfdb7869dfca4bcb3fea6a8cd34aafad2d6960"}
Jul 29 22:52:53.213172 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.213123    2581 scope.go:117] "RemoveContainer" containerID="12cbad704322c61f7af365e384e7987e02b36bccb522f1280243640e2ca5acb9"
Jul 29 22:52:53.220434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.220346    2581 scope.go:117] "RemoveContainer" containerID="cc6e834a11965a340a370b0000ecd732835b1ab2ab23486b8b744f5d173c3178"
Jul 29 22:52:53.226926 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.226911    2581 scope.go:117] "RemoveContainer" containerID="ebb99a544e5efe2ba5aae35d5f786ad982d3f4ea73e655dcae06be1f6564ed05"
Jul 29 22:52:53.233139 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.233123    2581 scope.go:117] "RemoveContainer" containerID="387642b8a776d5e4b8c2b478a91bfc5275846447e511263b70bc225207de6838"
Jul 29 22:52:53.237001 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.236980    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 22:52:53.240952 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.240930    2581 scope.go:117] "RemoveContainer" containerID="1ba7b1b023ce7ccac5a7376d2c7ee342a37ba6ba25a23e370299e40367fa9cf4"
Jul 29 22:52:53.241040 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.241017    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 22:52:53.247284 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.247269    2581 scope.go:117] "RemoveContainer" containerID="c1bf3521ed0e6dec36e55ab3bf3513ac3baf01bda17baf066e8f75d60f47cf17"
Jul 29 22:52:53.253994 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.253980    2581 scope.go:117] "RemoveContainer" containerID="0213eda7c803c6ccac30b8ad74e0fca65e27d083cd95c2a93ac14710a8ddd123"
Jul 29 22:52:53.260574 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.260556    2581 scope.go:117] "RemoveContainer" containerID="12cbad704322c61f7af365e384e7987e02b36bccb522f1280243640e2ca5acb9"
Jul 29 22:52:53.260851 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:52:53.260829    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12cbad704322c61f7af365e384e7987e02b36bccb522f1280243640e2ca5acb9\": container with ID starting with 12cbad704322c61f7af365e384e7987e02b36bccb522f1280243640e2ca5acb9 not found: ID does not exist" containerID="12cbad704322c61f7af365e384e7987e02b36bccb522f1280243640e2ca5acb9"
Jul 29 22:52:53.260896 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.260862    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12cbad704322c61f7af365e384e7987e02b36bccb522f1280243640e2ca5acb9"} err="failed to get container status \"12cbad704322c61f7af365e384e7987e02b36bccb522f1280243640e2ca5acb9\": rpc error: code = NotFound desc = could not find container \"12cbad704322c61f7af365e384e7987e02b36bccb522f1280243640e2ca5acb9\": container with ID starting with 12cbad704322c61f7af365e384e7987e02b36bccb522f1280243640e2ca5acb9 not found: ID does not exist"
Jul 29 22:52:53.260896 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.260879    2581 scope.go:117] "RemoveContainer" containerID="cc6e834a11965a340a370b0000ecd732835b1ab2ab23486b8b744f5d173c3178"
Jul 29 22:52:53.261128 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:52:53.261111    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc6e834a11965a340a370b0000ecd732835b1ab2ab23486b8b744f5d173c3178\": container with ID starting with cc6e834a11965a340a370b0000ecd732835b1ab2ab23486b8b744f5d173c3178 not found: ID does not exist" containerID="cc6e834a11965a340a370b0000ecd732835b1ab2ab23486b8b744f5d173c3178"
Jul 29 22:52:53.261163 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.261134    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc6e834a11965a340a370b0000ecd732835b1ab2ab23486b8b744f5d173c3178"} err="failed to get container status \"cc6e834a11965a340a370b0000ecd732835b1ab2ab23486b8b744f5d173c3178\": rpc error: code = NotFound desc = could not find container \"cc6e834a11965a340a370b0000ecd732835b1ab2ab23486b8b744f5d173c3178\": container with ID starting with cc6e834a11965a340a370b0000ecd732835b1ab2ab23486b8b744f5d173c3178 not found: ID does not exist"
Jul 29 22:52:53.261163 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.261149    2581 scope.go:117] "RemoveContainer" containerID="ebb99a544e5efe2ba5aae35d5f786ad982d3f4ea73e655dcae06be1f6564ed05"
Jul 29 22:52:53.261393 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:52:53.261376    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb99a544e5efe2ba5aae35d5f786ad982d3f4ea73e655dcae06be1f6564ed05\": container with ID starting with ebb99a544e5efe2ba5aae35d5f786ad982d3f4ea73e655dcae06be1f6564ed05 not found: ID does not exist" containerID="ebb99a544e5efe2ba5aae35d5f786ad982d3f4ea73e655dcae06be1f6564ed05"
Jul 29 22:52:53.261427 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.261400    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb99a544e5efe2ba5aae35d5f786ad982d3f4ea73e655dcae06be1f6564ed05"} err="failed to get container status \"ebb99a544e5efe2ba5aae35d5f786ad982d3f4ea73e655dcae06be1f6564ed05\": rpc error: code = NotFound desc = could not find container \"ebb99a544e5efe2ba5aae35d5f786ad982d3f4ea73e655dcae06be1f6564ed05\": container with ID starting with ebb99a544e5efe2ba5aae35d5f786ad982d3f4ea73e655dcae06be1f6564ed05 not found: ID does not exist"
Jul 29 22:52:53.261427 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.261415    2581 scope.go:117] "RemoveContainer" containerID="387642b8a776d5e4b8c2b478a91bfc5275846447e511263b70bc225207de6838"
Jul 29 22:52:53.261635 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:52:53.261619    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"387642b8a776d5e4b8c2b478a91bfc5275846447e511263b70bc225207de6838\": container with ID starting with 387642b8a776d5e4b8c2b478a91bfc5275846447e511263b70bc225207de6838 not found: ID does not exist" containerID="387642b8a776d5e4b8c2b478a91bfc5275846447e511263b70bc225207de6838"
Jul 29 22:52:53.261671 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.261640    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"387642b8a776d5e4b8c2b478a91bfc5275846447e511263b70bc225207de6838"} err="failed to get container status \"387642b8a776d5e4b8c2b478a91bfc5275846447e511263b70bc225207de6838\": rpc error: code = NotFound desc = could not find container \"387642b8a776d5e4b8c2b478a91bfc5275846447e511263b70bc225207de6838\": container with ID starting with 387642b8a776d5e4b8c2b478a91bfc5275846447e511263b70bc225207de6838 not found: ID does not exist"
Jul 29 22:52:53.261671 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.261654    2581 scope.go:117] "RemoveContainer" containerID="1ba7b1b023ce7ccac5a7376d2c7ee342a37ba6ba25a23e370299e40367fa9cf4"
Jul 29 22:52:53.261883 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:52:53.261864    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba7b1b023ce7ccac5a7376d2c7ee342a37ba6ba25a23e370299e40367fa9cf4\": container with ID starting with 1ba7b1b023ce7ccac5a7376d2c7ee342a37ba6ba25a23e370299e40367fa9cf4 not found: ID does not exist" containerID="1ba7b1b023ce7ccac5a7376d2c7ee342a37ba6ba25a23e370299e40367fa9cf4"
Jul 29 22:52:53.261962 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.261890    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba7b1b023ce7ccac5a7376d2c7ee342a37ba6ba25a23e370299e40367fa9cf4"} err="failed to get container status \"1ba7b1b023ce7ccac5a7376d2c7ee342a37ba6ba25a23e370299e40367fa9cf4\": rpc error: code = NotFound desc = could not find container \"1ba7b1b023ce7ccac5a7376d2c7ee342a37ba6ba25a23e370299e40367fa9cf4\": container with ID starting with 1ba7b1b023ce7ccac5a7376d2c7ee342a37ba6ba25a23e370299e40367fa9cf4 not found: ID does not exist"
Jul 29 22:52:53.261962 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.261909    2581 scope.go:117] "RemoveContainer" containerID="c1bf3521ed0e6dec36e55ab3bf3513ac3baf01bda17baf066e8f75d60f47cf17"
Jul 29 22:52:53.262137 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:52:53.262121    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1bf3521ed0e6dec36e55ab3bf3513ac3baf01bda17baf066e8f75d60f47cf17\": container with ID starting with c1bf3521ed0e6dec36e55ab3bf3513ac3baf01bda17baf066e8f75d60f47cf17 not found: ID does not exist" containerID="c1bf3521ed0e6dec36e55ab3bf3513ac3baf01bda17baf066e8f75d60f47cf17"
Jul 29 22:52:53.262175 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.262142    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1bf3521ed0e6dec36e55ab3bf3513ac3baf01bda17baf066e8f75d60f47cf17"} err="failed to get container status \"c1bf3521ed0e6dec36e55ab3bf3513ac3baf01bda17baf066e8f75d60f47cf17\": rpc error: code = NotFound desc = could not find container \"c1bf3521ed0e6dec36e55ab3bf3513ac3baf01bda17baf066e8f75d60f47cf17\": container with ID starting with c1bf3521ed0e6dec36e55ab3bf3513ac3baf01bda17baf066e8f75d60f47cf17 not found: ID does not exist"
Jul 29 22:52:53.262175 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.262155    2581 scope.go:117] "RemoveContainer" containerID="0213eda7c803c6ccac30b8ad74e0fca65e27d083cd95c2a93ac14710a8ddd123"
Jul 29 22:52:53.262374 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:52:53.262358    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0213eda7c803c6ccac30b8ad74e0fca65e27d083cd95c2a93ac14710a8ddd123\": container with ID starting with 0213eda7c803c6ccac30b8ad74e0fca65e27d083cd95c2a93ac14710a8ddd123 not found: ID does not exist" containerID="0213eda7c803c6ccac30b8ad74e0fca65e27d083cd95c2a93ac14710a8ddd123"
Jul 29 22:52:53.262415 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.262378    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0213eda7c803c6ccac30b8ad74e0fca65e27d083cd95c2a93ac14710a8ddd123"} err="failed to get container status \"0213eda7c803c6ccac30b8ad74e0fca65e27d083cd95c2a93ac14710a8ddd123\": rpc error: code = NotFound desc = could not find container \"0213eda7c803c6ccac30b8ad74e0fca65e27d083cd95c2a93ac14710a8ddd123\": container with ID starting with 0213eda7c803c6ccac30b8ad74e0fca65e27d083cd95c2a93ac14710a8ddd123 not found: ID does not exist"
Jul 29 22:52:53.262415 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.262390    2581 scope.go:117] "RemoveContainer" containerID="12cbad704322c61f7af365e384e7987e02b36bccb522f1280243640e2ca5acb9"
Jul 29 22:52:53.262572 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.262557    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12cbad704322c61f7af365e384e7987e02b36bccb522f1280243640e2ca5acb9"} err="failed to get container status \"12cbad704322c61f7af365e384e7987e02b36bccb522f1280243640e2ca5acb9\": rpc error: code = NotFound desc = could not find container \"12cbad704322c61f7af365e384e7987e02b36bccb522f1280243640e2ca5acb9\": container with ID starting with 12cbad704322c61f7af365e384e7987e02b36bccb522f1280243640e2ca5acb9 not found: ID does not exist"
Jul 29 22:52:53.262615 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.262572    2581 scope.go:117] "RemoveContainer" containerID="cc6e834a11965a340a370b0000ecd732835b1ab2ab23486b8b744f5d173c3178"
Jul 29 22:52:53.262769 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.262749    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc6e834a11965a340a370b0000ecd732835b1ab2ab23486b8b744f5d173c3178"} err="failed to get container status \"cc6e834a11965a340a370b0000ecd732835b1ab2ab23486b8b744f5d173c3178\": rpc error: code = NotFound desc = could not find container \"cc6e834a11965a340a370b0000ecd732835b1ab2ab23486b8b744f5d173c3178\": container with ID starting with cc6e834a11965a340a370b0000ecd732835b1ab2ab23486b8b744f5d173c3178 not found: ID does not exist"
Jul 29 22:52:53.262838 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.262770    2581 scope.go:117] "RemoveContainer" containerID="ebb99a544e5efe2ba5aae35d5f786ad982d3f4ea73e655dcae06be1f6564ed05"
Jul 29 22:52:53.263010 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.262994    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb99a544e5efe2ba5aae35d5f786ad982d3f4ea73e655dcae06be1f6564ed05"} err="failed to get container status \"ebb99a544e5efe2ba5aae35d5f786ad982d3f4ea73e655dcae06be1f6564ed05\": rpc error: code = NotFound desc = could not find container \"ebb99a544e5efe2ba5aae35d5f786ad982d3f4ea73e655dcae06be1f6564ed05\": container with ID starting with ebb99a544e5efe2ba5aae35d5f786ad982d3f4ea73e655dcae06be1f6564ed05 not found: ID does not exist"
Jul 29 22:52:53.263054 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.263010    2581 scope.go:117] "RemoveContainer" containerID="387642b8a776d5e4b8c2b478a91bfc5275846447e511263b70bc225207de6838"
Jul 29 22:52:53.263196 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.263179    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"387642b8a776d5e4b8c2b478a91bfc5275846447e511263b70bc225207de6838"} err="failed to get container status \"387642b8a776d5e4b8c2b478a91bfc5275846447e511263b70bc225207de6838\": rpc error: code = NotFound desc = could not find container \"387642b8a776d5e4b8c2b478a91bfc5275846447e511263b70bc225207de6838\": container with ID starting with 387642b8a776d5e4b8c2b478a91bfc5275846447e511263b70bc225207de6838 not found: ID does not exist"
Jul 29 22:52:53.263196 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.263195    2581 scope.go:117] "RemoveContainer" containerID="1ba7b1b023ce7ccac5a7376d2c7ee342a37ba6ba25a23e370299e40367fa9cf4"
Jul 29 22:52:53.263391 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.263375    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba7b1b023ce7ccac5a7376d2c7ee342a37ba6ba25a23e370299e40367fa9cf4"} err="failed to get container status \"1ba7b1b023ce7ccac5a7376d2c7ee342a37ba6ba25a23e370299e40367fa9cf4\": rpc error: code = NotFound desc = could not find container \"1ba7b1b023ce7ccac5a7376d2c7ee342a37ba6ba25a23e370299e40367fa9cf4\": container with ID starting with 1ba7b1b023ce7ccac5a7376d2c7ee342a37ba6ba25a23e370299e40367fa9cf4 not found: ID does not exist"
Jul 29 22:52:53.263434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.263392    2581 scope.go:117] "RemoveContainer" containerID="c1bf3521ed0e6dec36e55ab3bf3513ac3baf01bda17baf066e8f75d60f47cf17"
Jul 29 22:52:53.263553 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.263535    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1bf3521ed0e6dec36e55ab3bf3513ac3baf01bda17baf066e8f75d60f47cf17"} err="failed to get container status \"c1bf3521ed0e6dec36e55ab3bf3513ac3baf01bda17baf066e8f75d60f47cf17\": rpc error: code = NotFound desc = could not find container \"c1bf3521ed0e6dec36e55ab3bf3513ac3baf01bda17baf066e8f75d60f47cf17\": container with ID starting with c1bf3521ed0e6dec36e55ab3bf3513ac3baf01bda17baf066e8f75d60f47cf17 not found: ID does not exist"
Jul 29 22:52:53.263592 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.263553    2581 scope.go:117] "RemoveContainer" containerID="0213eda7c803c6ccac30b8ad74e0fca65e27d083cd95c2a93ac14710a8ddd123"
Jul 29 22:52:53.263734 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.263720    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0213eda7c803c6ccac30b8ad74e0fca65e27d083cd95c2a93ac14710a8ddd123"} err="failed to get container status \"0213eda7c803c6ccac30b8ad74e0fca65e27d083cd95c2a93ac14710a8ddd123\": rpc error: code = NotFound desc = could not find container \"0213eda7c803c6ccac30b8ad74e0fca65e27d083cd95c2a93ac14710a8ddd123\": container with ID starting with 0213eda7c803c6ccac30b8ad74e0fca65e27d083cd95c2a93ac14710a8ddd123 not found: ID does not exist"
Jul 29 22:52:53.267266 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267248    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 22:52:53.267515 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267505    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="kube-rbac-proxy-metric"
Jul 29 22:52:53.267556 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267517    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="kube-rbac-proxy-metric"
Jul 29 22:52:53.267556 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267527    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="config-reloader"
Jul 29 22:52:53.267556 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267534    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="config-reloader"
Jul 29 22:52:53.267556 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267545    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="init-config-reloader"
Jul 29 22:52:53.267556 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267551    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="init-config-reloader"
Jul 29 22:52:53.267697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267557    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="kube-rbac-proxy"
Jul 29 22:52:53.267697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267563    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="kube-rbac-proxy"
Jul 29 22:52:53.267697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267571    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="prom-label-proxy"
Jul 29 22:52:53.267697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267577    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="prom-label-proxy"
Jul 29 22:52:53.267697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267585    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7e081dc-a705-4255-aed8-c20f7090c5b3" containerName="registry"
Jul 29 22:52:53.267697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267590    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7e081dc-a705-4255-aed8-c20f7090c5b3" containerName="registry"
Jul 29 22:52:53.267697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267595    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="kube-rbac-proxy-web"
Jul 29 22:52:53.267697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267600    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="kube-rbac-proxy-web"
Jul 29 22:52:53.267697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267608    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="alertmanager"
Jul 29 22:52:53.267697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267613    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="alertmanager"
Jul 29 22:52:53.267697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267660    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="kube-rbac-proxy"
Jul 29 22:52:53.267697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267668    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="kube-rbac-proxy-metric"
Jul 29 22:52:53.267697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267673    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="alertmanager"
Jul 29 22:52:53.267697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267678    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7e081dc-a705-4255-aed8-c20f7090c5b3" containerName="registry"
Jul 29 22:52:53.267697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267682    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="prom-label-proxy"
Jul 29 22:52:53.267697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267688    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="kube-rbac-proxy-web"
Jul 29 22:52:53.267697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.267694    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" containerName="config-reloader"
Jul 29 22:52:53.273756 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.273739    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.276925 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.276908    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vcndz\""
Jul 29 22:52:53.277525 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.277505    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 29 22:52:53.277525 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.277518    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 29 22:52:53.277689 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.277518    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 29 22:52:53.277776 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.277762    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 29 22:52:53.277852 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.277828    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 29 22:52:53.277974 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.277955    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 29 22:52:53.279018 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.278998    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 29 22:52:53.279338 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.279321    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 29 22:52:53.284487 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.284468    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 29 22:52:53.285479 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.285459    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 22:52:53.291251 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.291230    2581 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-config-volume\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:53.291320 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.291256    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-metrics-client-ca\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:53.291320 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.291272    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-main-tls\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:53.291320 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.291286    2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-config-out\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:53.291320 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.291299    2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-tls-assets\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:53.291320 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.291310    2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-web-config\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:53.291473 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.291325    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:53.291473 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.291340    2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-alertmanager-main-db\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:53.291473 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.291353    2581 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-cluster-tls-config\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:53.291473 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.291365    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnzcn\" (UniqueName: \"kubernetes.io/projected/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-kube-api-access-wnzcn\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:53.291473 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.291380    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:53.291473 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.291395    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ffa1afb-7773-4b55-b5d3-e46864abf5ff-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:53.392143 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.392108    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-web-config\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.392143 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.392140    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6dca323-830f-4221-837c-e4ddfb530361-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.392381 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.392158    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-config-volume\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.392381 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.392213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f6dca323-830f-4221-837c-e4ddfb530361-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.392381 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.392262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.392381 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.392310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.392381 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.392346    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6dca323-830f-4221-837c-e4ddfb530361-config-out\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.392381 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.392372    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmqmp\" (UniqueName: \"kubernetes.io/projected/f6dca323-830f-4221-837c-e4ddfb530361-kube-api-access-wmqmp\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.392602 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.392408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.392602 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.392426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.392602 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.392448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6dca323-830f-4221-837c-e4ddfb530361-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.392602 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.392468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.392602 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.392484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6dca323-830f-4221-837c-e4ddfb530361-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.492924 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.492839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.492924 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.492869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.492924 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.492888    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6dca323-830f-4221-837c-e4ddfb530361-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.492924 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.492907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.493236 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.492950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6dca323-830f-4221-837c-e4ddfb530361-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.493236 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.493030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-web-config\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.493236 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.493056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6dca323-830f-4221-837c-e4ddfb530361-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.493236 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.493081    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-config-volume\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.493236 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.493119    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f6dca323-830f-4221-837c-e4ddfb530361-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.493236 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.493147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.493236 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.493201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.493236 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.493228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6dca323-830f-4221-837c-e4ddfb530361-config-out\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.493602 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.493253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmqmp\" (UniqueName: \"kubernetes.io/projected/f6dca323-830f-4221-837c-e4ddfb530361-kube-api-access-wmqmp\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.494136 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.493767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6dca323-830f-4221-837c-e4ddfb530361-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.494136 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.493863    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6dca323-830f-4221-837c-e4ddfb530361-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.496045 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.496013    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.496193 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.496172    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.496274 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.496179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-web-config\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.496274 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.496206    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.496274 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.496211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-config-volume\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.496443 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.496392    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.496443 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.496427    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f6dca323-830f-4221-837c-e4ddfb530361-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.496550 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.496516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f6dca323-830f-4221-837c-e4ddfb530361-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.496550 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.496546    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6dca323-830f-4221-837c-e4ddfb530361-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.497773 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.497753    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6dca323-830f-4221-837c-e4ddfb530361-config-out\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.501004 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.500979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmqmp\" (UniqueName: \"kubernetes.io/projected/f6dca323-830f-4221-837c-e4ddfb530361-kube-api-access-wmqmp\") pod \"alertmanager-main-0\" (UID: \"f6dca323-830f-4221-837c-e4ddfb530361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.583151 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.583129    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 22:52:53.716294 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.716271    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 22:52:53.717880 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:52:53.717857    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6dca323_830f_4221_837c_e4ddfb530361.slice/crio-1f288b9d2434be1c6c18d2829d2e2d2a4936c2c350a5bde803efa2c44097cb91 WatchSource:0}: Error finding container 1f288b9d2434be1c6c18d2829d2e2d2a4936c2c350a5bde803efa2c44097cb91: Status 404 returned error can't find the container with id 1f288b9d2434be1c6c18d2829d2e2d2a4936c2c350a5bde803efa2c44097cb91
Jul 29 22:52:53.951314 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:53.951284    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ffa1afb-7773-4b55-b5d3-e46864abf5ff" path="/var/lib/kubelet/pods/6ffa1afb-7773-4b55-b5d3-e46864abf5ff/volumes"
Jul 29 22:52:54.218309 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:54.218277    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6dca323-830f-4221-837c-e4ddfb530361" containerID="440227ee483787fc0fb4fb1d14cc2955028dac3163d27ee14669eaa39a279702" exitCode=0
Jul 29 22:52:54.218309 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:54.218311    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6dca323-830f-4221-837c-e4ddfb530361","Type":"ContainerDied","Data":"440227ee483787fc0fb4fb1d14cc2955028dac3163d27ee14669eaa39a279702"}
Jul 29 22:52:54.218728 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:54.218359    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6dca323-830f-4221-837c-e4ddfb530361","Type":"ContainerStarted","Data":"1f288b9d2434be1c6c18d2829d2e2d2a4936c2c350a5bde803efa2c44097cb91"}
Jul 29 22:52:55.224985 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:55.224952    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6dca323-830f-4221-837c-e4ddfb530361","Type":"ContainerStarted","Data":"61e1a3cc006c33b7ad784c228020ce2e14ac5ae0ec7bed8c07dae05da9553e49"}
Jul 29 22:52:55.225356 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:55.225015    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6dca323-830f-4221-837c-e4ddfb530361","Type":"ContainerStarted","Data":"e103ed6c4a5e18d55233f15466b4c9afcd1a78b822c14bc72c7b3bb7146a6256"}
Jul 29 22:52:55.225356 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:55.225042    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6dca323-830f-4221-837c-e4ddfb530361","Type":"ContainerStarted","Data":"657c46c2db04d5d30484315ef4cca4250ba2a6e5db391a94895e8e68a1024fe3"}
Jul 29 22:52:55.225356 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:55.225065    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6dca323-830f-4221-837c-e4ddfb530361","Type":"ContainerStarted","Data":"d787d2231b2e154f4659c248919e03d0d1c5f48336a9fd8686eadc06b1811ffb"}
Jul 29 22:52:55.225356 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:55.225088    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6dca323-830f-4221-837c-e4ddfb530361","Type":"ContainerStarted","Data":"58b25346fb2ae01ac6e34e098d1e6745d10b1454002adeea819f3670e177f1c5"}
Jul 29 22:52:55.225356 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:55.225112    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6dca323-830f-4221-837c-e4ddfb530361","Type":"ContainerStarted","Data":"53b89663a00048ae943947b4612a2e68279746d4487b596fe8af1e5f5f875607"}
Jul 29 22:52:55.254533 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:55.254485    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.24104904 podStartE2EDuration="2.254470356s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.421316ms" isStatefulPod=false podCreationTimestamp="2026-07-29 22:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:52:55.25130947 +0000 UTC m=+397.809654287" watchObservedRunningTime="2026-07-29 22:52:55.254470356 +0000 UTC m=+397.812815172"
Jul 29 22:52:56.018303 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.018215    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 22:52:56.018511 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.018474    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="prometheus" containerID="cri-o://9c37354c6f76e060e30744afd27f2ba020170e5c4ab9929697280b063b2af572" gracePeriod=600
Jul 29 22:52:56.018511 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.018501    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="thanos-sidecar" containerID="cri-o://d1fb2e40c4e562f882abb1140720382fe82175951d06df8939362f4e8aef2ef4" gracePeriod=600
Jul 29 22:52:56.018697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.018530    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="config-reloader" containerID="cri-o://6012c5348b1151b23e9b0f98584e3a219c5b2d1aeb923960f080a8cf8a5a0ab6" gracePeriod=600
Jul 29 22:52:56.018697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.018482    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="kube-rbac-proxy" containerID="cri-o://e975fa7ff4d6bf7d2b33d812a845714435b953840401d15a3609bf10f015db0d" gracePeriod=600
Jul 29 22:52:56.018697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.018565    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="kube-rbac-proxy-web" containerID="cri-o://663bfa3e16bc9ea350a0c823979217c5369897173096c1995fd5c98398f3a6af" gracePeriod=600
Jul 29 22:52:56.018697 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.018527    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b9eb7c26626169f0e224062af0252b8dd86bebb447ab48814a9f2d8224d0d239" gracePeriod=600
Jul 29 22:52:56.237916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.237877    2581 generic.go:356] "Generic (PLEG): container finished" podID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerID="b9eb7c26626169f0e224062af0252b8dd86bebb447ab48814a9f2d8224d0d239" exitCode=0
Jul 29 22:52:56.237916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.237908    2581 generic.go:356] "Generic (PLEG): container finished" podID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerID="e975fa7ff4d6bf7d2b33d812a845714435b953840401d15a3609bf10f015db0d" exitCode=0
Jul 29 22:52:56.237916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.237917    2581 generic.go:356] "Generic (PLEG): container finished" podID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerID="663bfa3e16bc9ea350a0c823979217c5369897173096c1995fd5c98398f3a6af" exitCode=0
Jul 29 22:52:56.237916 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.237925    2581 generic.go:356] "Generic (PLEG): container finished" podID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerID="d1fb2e40c4e562f882abb1140720382fe82175951d06df8939362f4e8aef2ef4" exitCode=0
Jul 29 22:52:56.238528 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.237933    2581 generic.go:356] "Generic (PLEG): container finished" podID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerID="6012c5348b1151b23e9b0f98584e3a219c5b2d1aeb923960f080a8cf8a5a0ab6" exitCode=0
Jul 29 22:52:56.238528 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.237942    2581 generic.go:356] "Generic (PLEG): container finished" podID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerID="9c37354c6f76e060e30744afd27f2ba020170e5c4ab9929697280b063b2af572" exitCode=0
Jul 29 22:52:56.238528 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.237943    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84e9f303-8aa4-4610-be99-1c8ce985cf7c","Type":"ContainerDied","Data":"b9eb7c26626169f0e224062af0252b8dd86bebb447ab48814a9f2d8224d0d239"}
Jul 29 22:52:56.238528 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.238014    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84e9f303-8aa4-4610-be99-1c8ce985cf7c","Type":"ContainerDied","Data":"e975fa7ff4d6bf7d2b33d812a845714435b953840401d15a3609bf10f015db0d"}
Jul 29 22:52:56.238528 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.238041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84e9f303-8aa4-4610-be99-1c8ce985cf7c","Type":"ContainerDied","Data":"663bfa3e16bc9ea350a0c823979217c5369897173096c1995fd5c98398f3a6af"}
Jul 29 22:52:56.238528 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.238075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84e9f303-8aa4-4610-be99-1c8ce985cf7c","Type":"ContainerDied","Data":"d1fb2e40c4e562f882abb1140720382fe82175951d06df8939362f4e8aef2ef4"}
Jul 29 22:52:56.238528 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.238101    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84e9f303-8aa4-4610-be99-1c8ce985cf7c","Type":"ContainerDied","Data":"6012c5348b1151b23e9b0f98584e3a219c5b2d1aeb923960f080a8cf8a5a0ab6"}
Jul 29 22:52:56.238528 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.238124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84e9f303-8aa4-4610-be99-1c8ce985cf7c","Type":"ContainerDied","Data":"9c37354c6f76e060e30744afd27f2ba020170e5c4ab9929697280b063b2af572"}
Jul 29 22:52:56.267429 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.267405    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:56.282746 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.282682    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:52:56.417079 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417038    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84e9f303-8aa4-4610-be99-1c8ce985cf7c-tls-assets\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417079 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417084    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-configmap-serving-certs-ca-bundle\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417299 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417105    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-thanos-prometheus-http-client-file\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417299 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417125    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-grpc-tls\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417429 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417301    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-config\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417429 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417348    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-trusted-ca-bundle\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417429 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417375    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417429 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417424    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-configmap-kubelet-serving-ca-bundle\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417613 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417430    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:52:56.417613 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417449    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-rulefiles-2\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417613 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417474    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-configmap-metrics-client-ca\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417613 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417505    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-prometheus-k8s-tls\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417613 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417542    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-metrics-client-certs\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417613 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417583    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-db\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417923 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417619    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-rulefiles-0\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417923 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417658    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417923 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417688    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84e9f303-8aa4-4610-be99-1c8ce985cf7c-config-out\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417923 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417720    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-web-config\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417923 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417734    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:52:56.417923 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417742    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:52:56.417923 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417775    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfgcc\" (UniqueName: \"kubernetes.io/projected/84e9f303-8aa4-4610-be99-1c8ce985cf7c-kube-api-access-qfgcc\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417923 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417823    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-rulefiles-1\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.417923 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.417850    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-kube-rbac-proxy\") pod \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\" (UID: \"84e9f303-8aa4-4610-be99-1c8ce985cf7c\") "
Jul 29 22:52:56.418344 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.418261    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:52:56.418344 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.418308    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.418344 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.418333    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.418568 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.418356    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.418617 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.418578    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:52:56.419594 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.419055    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:52:56.420682 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.420103    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:52:56.420682 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.420556    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:52:56.421448 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.421411    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:56.421783 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.421762    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:56.422270 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.422215    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:56.422368 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.422273    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-config" (OuterVolumeSpecName: "config") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:56.422368 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.422292    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e9f303-8aa4-4610-be99-1c8ce985cf7c-config-out" (OuterVolumeSpecName: "config-out") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:52:56.422368 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.422318    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:56.422616 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.422470    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e9f303-8aa4-4610-be99-1c8ce985cf7c-kube-api-access-qfgcc" (OuterVolumeSpecName: "kube-api-access-qfgcc") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "kube-api-access-qfgcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:52:56.422851 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.422823    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:56.422942 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.422903    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e9f303-8aa4-4610-be99-1c8ce985cf7c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:52:56.423384 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.423368    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:56.423746 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.423724    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:56.431389 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.431371    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-web-config" (OuterVolumeSpecName: "web-config") pod "84e9f303-8aa4-4610-be99-1c8ce985cf7c" (UID: "84e9f303-8aa4-4610-be99-1c8ce985cf7c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:56.519105 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.519072    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.519105 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.519103    2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84e9f303-8aa4-4610-be99-1c8ce985cf7c-config-out\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.519105 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.519113    2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-web-config\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.519242 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.519121    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfgcc\" (UniqueName: \"kubernetes.io/projected/84e9f303-8aa4-4610-be99-1c8ce985cf7c-kube-api-access-qfgcc\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.519242 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.519131    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.519242 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.519140    2581 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-kube-rbac-proxy\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.519242 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.519148    2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84e9f303-8aa4-4610-be99-1c8ce985cf7c-tls-assets\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.519242 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.519157    2581 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.519242 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.519166    2581 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-grpc-tls\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.519242 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.519174    2581 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-config\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.519242 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.519182    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.519242 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.519190    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.519242 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.519199    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-configmap-metrics-client-ca\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.519242 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.519207    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.519242 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.519215    2581 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84e9f303-8aa4-4610-be99-1c8ce985cf7c-secret-metrics-client-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.519242 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.519223    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-db\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:56.519242 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:56.519231    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84e9f303-8aa4-4610-be99-1c8ce985cf7c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:52:57.246059 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.246018    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84e9f303-8aa4-4610-be99-1c8ce985cf7c","Type":"ContainerDied","Data":"a97e82b4139230a2eb2156e86f6b0d81ddfaff775fa73f748fe52c16d682c849"}
Jul 29 22:52:57.246059 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.246045    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.246545 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.246117    2581 scope.go:117] "RemoveContainer" containerID="b9eb7c26626169f0e224062af0252b8dd86bebb447ab48814a9f2d8224d0d239"
Jul 29 22:52:57.255325 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.255300    2581 scope.go:117] "RemoveContainer" containerID="e975fa7ff4d6bf7d2b33d812a845714435b953840401d15a3609bf10f015db0d"
Jul 29 22:52:57.263109 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.263090    2581 scope.go:117] "RemoveContainer" containerID="663bfa3e16bc9ea350a0c823979217c5369897173096c1995fd5c98398f3a6af"
Jul 29 22:52:57.269987 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.269965    2581 scope.go:117] "RemoveContainer" containerID="d1fb2e40c4e562f882abb1140720382fe82175951d06df8939362f4e8aef2ef4"
Jul 29 22:52:57.272946 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.272922    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 22:52:57.277527 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.277510    2581 scope.go:117] "RemoveContainer" containerID="6012c5348b1151b23e9b0f98584e3a219c5b2d1aeb923960f080a8cf8a5a0ab6"
Jul 29 22:52:57.283869 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.283847    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 22:52:57.284971 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.284957    2581 scope.go:117] "RemoveContainer" containerID="9c37354c6f76e060e30744afd27f2ba020170e5c4ab9929697280b063b2af572"
Jul 29 22:52:57.292055 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.292037    2581 scope.go:117] "RemoveContainer" containerID="6603d8ebc8461c8276c43a7323c31aafeddd1a46879105b07d9f4be68cd9b855"
Jul 29 22:52:57.310103 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310079    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 22:52:57.310384 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310370    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="init-config-reloader"
Jul 29 22:52:57.310433 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310385    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="init-config-reloader"
Jul 29 22:52:57.310433 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310401    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="kube-rbac-proxy-web"
Jul 29 22:52:57.310433 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310409    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="kube-rbac-proxy-web"
Jul 29 22:52:57.310433 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310425    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="prometheus"
Jul 29 22:52:57.310433 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310434    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="prometheus"
Jul 29 22:52:57.310598 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310445    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="config-reloader"
Jul 29 22:52:57.310598 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310453    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="config-reloader"
Jul 29 22:52:57.310598 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310465    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="kube-rbac-proxy"
Jul 29 22:52:57.310598 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310472    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="kube-rbac-proxy"
Jul 29 22:52:57.310598 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310484    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="thanos-sidecar"
Jul 29 22:52:57.310598 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310492    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="thanos-sidecar"
Jul 29 22:52:57.310598 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310498    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="kube-rbac-proxy-thanos"
Jul 29 22:52:57.310598 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310503    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="kube-rbac-proxy-thanos"
Jul 29 22:52:57.310598 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310568    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="config-reloader"
Jul 29 22:52:57.310598 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310576    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="kube-rbac-proxy"
Jul 29 22:52:57.310598 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310583    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="thanos-sidecar"
Jul 29 22:52:57.310598 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310589    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="prometheus"
Jul 29 22:52:57.310598 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310594    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="kube-rbac-proxy-thanos"
Jul 29 22:52:57.310598 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.310600    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" containerName="kube-rbac-proxy-web"
Jul 29 22:52:57.316856 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.316838    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.319886 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.319870    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 29 22:52:57.319976 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.319909    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 29 22:52:57.320339 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.320153    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 29 22:52:57.320339 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.320220    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 29 22:52:57.320339 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.320228    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 29 22:52:57.320339 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.320285    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 29 22:52:57.320339 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.320314    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 29 22:52:57.321053 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.321030    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 29 22:52:57.321237 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.321214    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 29 22:52:57.321423 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.321223    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 29 22:52:57.321494 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.321476    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4qh47\""
Jul 29 22:52:57.321629 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.321612    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7dl2bq2qmt4bd\""
Jul 29 22:52:57.325364 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.325178    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 29 22:52:57.326289 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.325494    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 29 22:52:57.326289 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.325677    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 29 22:52:57.330058 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.330028    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 29 22:52:57.330247 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.330229    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 22:52:57.331335 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.331303    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 29 22:52:57.427904 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.427874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-config\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.427904 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.427906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.428116 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.427931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.428116 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.427952    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.428116 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.428017    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4220a124-428a-4538-bb23-03a987e1a661-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.428116 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.428051    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.428116 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.428082    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.428116 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.428114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.428313 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.428137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.428313 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.428150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.428313 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.428166    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-web-config\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.428313 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.428191    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.428313 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.428231    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.428313 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.428253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.428313 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.428286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4220a124-428a-4538-bb23-03a987e1a661-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.428313 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.428309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4220a124-428a-4538-bb23-03a987e1a661-config-out\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.428558 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.428322    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.428558 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.428343    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.428558 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.428359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d22np\" (UniqueName: \"kubernetes.io/projected/4220a124-428a-4538-bb23-03a987e1a661-kube-api-access-d22np\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.428558 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.428381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.529200 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529158    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.529350 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.529350 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.529350 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.529350 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.529350 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-web-config\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.529350 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.529649 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.529649 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.529649 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4220a124-428a-4538-bb23-03a987e1a661-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.529649 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529463    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4220a124-428a-4538-bb23-03a987e1a661-config-out\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.529649 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.529649 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.529649 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d22np\" (UniqueName: \"kubernetes.io/projected/4220a124-428a-4538-bb23-03a987e1a661-kube-api-access-d22np\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.529649 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.529649 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-config\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.529649 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529632    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.530141 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529665    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.530141 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.530141 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.529733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4220a124-428a-4538-bb23-03a987e1a661-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.530141 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.530048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.530141 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.530096    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4220a124-428a-4538-bb23-03a987e1a661-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.530141 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.530130    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.530416 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.530226    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.533434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.532458    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.533434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.532482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.533434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.532709    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4220a124-428a-4538-bb23-03a987e1a661-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.533434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.532985    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.533434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.533045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.533434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.533100    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.533434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.533196    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-web-config\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.533434 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.533379    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.533869 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.533535    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.533869 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.533692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.534074 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.534053    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4220a124-428a-4538-bb23-03a987e1a661-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.535373 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.535320    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.535469 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.535440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4220a124-428a-4538-bb23-03a987e1a661-config-out\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.535611 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.535590    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.535779 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.535760    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4220a124-428a-4538-bb23-03a987e1a661-config\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.538507 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.538490    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d22np\" (UniqueName: \"kubernetes.io/projected/4220a124-428a-4538-bb23-03a987e1a661-kube-api-access-d22np\") pod \"prometheus-k8s-0\" (UID: \"4220a124-428a-4538-bb23-03a987e1a661\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.632703 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.632681    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:52:57.764089 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.764065    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 22:52:57.766461 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:52:57.766437    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4220a124_428a_4538_bb23_03a987e1a661.slice/crio-f76a3d47cf7c2c25c12f422f4aeb425a948e2c79efbf16fe0720d25f3cd85ae8 WatchSource:0}: Error finding container f76a3d47cf7c2c25c12f422f4aeb425a948e2c79efbf16fe0720d25f3cd85ae8: Status 404 returned error can't find the container with id f76a3d47cf7c2c25c12f422f4aeb425a948e2c79efbf16fe0720d25f3cd85ae8
Jul 29 22:52:57.952028 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:57.952001    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84e9f303-8aa4-4610-be99-1c8ce985cf7c" path="/var/lib/kubelet/pods/84e9f303-8aa4-4610-be99-1c8ce985cf7c/volumes"
Jul 29 22:52:58.252523 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:58.252498    2581 generic.go:356] "Generic (PLEG): container finished" podID="4220a124-428a-4538-bb23-03a987e1a661" containerID="02ea84729f0f738dc356cefb6f1796ad3dfc0e0c6651c1bd06f451a93d646da2" exitCode=0
Jul 29 22:52:58.252882 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:58.252564    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4220a124-428a-4538-bb23-03a987e1a661","Type":"ContainerDied","Data":"02ea84729f0f738dc356cefb6f1796ad3dfc0e0c6651c1bd06f451a93d646da2"}
Jul 29 22:52:58.252882 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:58.252616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4220a124-428a-4538-bb23-03a987e1a661","Type":"ContainerStarted","Data":"f76a3d47cf7c2c25c12f422f4aeb425a948e2c79efbf16fe0720d25f3cd85ae8"}
Jul 29 22:52:59.221317 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:59.221278    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:52:59.260868 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:59.260835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4220a124-428a-4538-bb23-03a987e1a661","Type":"ContainerStarted","Data":"d66cd7935fa131f9bae1553468ed0472725c0405b2daea8286a57c15ae76cbb3"}
Jul 29 22:52:59.261236 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:59.260930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4220a124-428a-4538-bb23-03a987e1a661","Type":"ContainerStarted","Data":"c9fe5656edb23e0e1f14eb81eb5a3cf4425ed4bf12efeb0754bfe1ffd3f371e2"}
Jul 29 22:52:59.261236 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:59.260957    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4220a124-428a-4538-bb23-03a987e1a661","Type":"ContainerStarted","Data":"1d411be476b88e72cb32963b0efc25ff39bccd4229d276c98bfec14833014d1a"}
Jul 29 22:52:59.261236 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:59.260983    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4220a124-428a-4538-bb23-03a987e1a661","Type":"ContainerStarted","Data":"6cfd21ae184861c97a47a6be43dbe0e2518e15990361b15de2d53f5640c538aa"}
Jul 29 22:52:59.261236 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:59.261007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4220a124-428a-4538-bb23-03a987e1a661","Type":"ContainerStarted","Data":"68a8ab0a703df8abc662b015efa521dd59b6d5317fde2f10b8c5b4eb867f5a0c"}
Jul 29 22:52:59.261236 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:59.261030    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4220a124-428a-4538-bb23-03a987e1a661","Type":"ContainerStarted","Data":"c2a73fef30b3c9f42b4e3549436cbf805a6931a7eb3d2e81d7a49656f7a35d17"}
Jul 29 22:52:59.296331 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:59.296285    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.27522001 podStartE2EDuration="2.29626854s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.04853ms" isStatefulPod=false podCreationTimestamp="2026-07-29 22:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:52:59.294374368 +0000 UTC m=+401.852719203" watchObservedRunningTime="2026-07-29 22:52:59.29626854 +0000 UTC m=+401.854613356"
Jul 29 22:52:59.302850 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:52:59.302815    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:53:01.174995 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:01.174951    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:53:01.727532 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:53:01.727490    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv" podUID="2fe2a7d4-0e16-4f28-9173-d5765b53e33b"
Jul 29 22:53:02.277234 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:02.277205    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:53:02.633164 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:02.633138    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:53:03.878419 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:03.878384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sk5qv\" (UID: \"2fe2a7d4-0e16-4f28-9173-d5765b53e33b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:53:03.880732 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:03.880708    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2fe2a7d4-0e16-4f28-9173-d5765b53e33b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sk5qv\" (UID: \"2fe2a7d4-0e16-4f28-9173-d5765b53e33b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:53:04.080530 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:04.080499    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-trcnk\""
Jul 29 22:53:04.087758 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:04.087735    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"
Jul 29 22:53:04.214440 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:04.214415    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv"]
Jul 29 22:53:04.217063 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:53:04.217038    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe2a7d4_0e16_4f28_9173_d5765b53e33b.slice/crio-4af0bc83e70b8de4bd81897839faa6b732cbcb38ed560604e5d651d57d013539 WatchSource:0}: Error finding container 4af0bc83e70b8de4bd81897839faa6b732cbcb38ed560604e5d651d57d013539: Status 404 returned error can't find the container with id 4af0bc83e70b8de4bd81897839faa6b732cbcb38ed560604e5d651d57d013539
Jul 29 22:53:04.285545 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:04.285511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv" event={"ID":"2fe2a7d4-0e16-4f28-9173-d5765b53e33b","Type":"ContainerStarted","Data":"4af0bc83e70b8de4bd81897839faa6b732cbcb38ed560604e5d651d57d013539"}
Jul 29 22:53:06.295391 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:06.295352    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv" event={"ID":"2fe2a7d4-0e16-4f28-9173-d5765b53e33b","Type":"ContainerStarted","Data":"47c9b210458b4e62895d30ba00d08b1ddda1c63551ae8d31f3823026a1208c87"}
Jul 29 22:53:06.315756 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:06.315709    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sk5qv" podStartSLOduration=381.952413359 podStartE2EDuration="6m23.315695371s" totalImagesPullingTime="1.363282012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:53:06.314388797 +0000 UTC m=+408.872733615" watchObservedRunningTime="2026-07-29 22:53:06.315695371 +0000 UTC m=+408.874040186"
Jul 29 22:53:06.436360 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:06.436326    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:53:06.572489 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:06.572403    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:53:06.751273 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:06.751244    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:53:07.066485 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:07.066455    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:53:07.167249 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:07.167222    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:53:07.221105 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:07.221081    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:53:10.627175 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:10.627126    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:53:10.729546 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:10.729513    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:53:10.786123 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:10.786086    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:53:24.728855 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:24.728814    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:53:24.782513 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:24.782484    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:53:57.633878 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:57.633842    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:53:57.649614 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:57.649590    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:53:58.517405 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:53:58.517377    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 22:54:17.009383 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:17.009349    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:17.070960 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:17.070927    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:17.365875 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:17.365837    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:17.469565 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:17.469510    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:17.568545 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:17.568508    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:39.459376 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:39.459338    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:39.556255 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:39.556222    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:44.888124 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:44.888084    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:45.165877 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:45.165786    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:45.267971 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:45.267946    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:45.388605 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:45.388567    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:45.726036 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:45.726006    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:45.856439 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:45.856408    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:45.963515 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:45.963487    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:46.062564 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:46.062465    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:46.253876 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:46.253841    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:46.440563 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:46.440535    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:46.642203 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:46.642171    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:46.743155 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:46.743069    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:54:50.275580 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:54:50.275543    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:55:20.028736 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:55:20.028649    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:55:31.275561 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:55:31.275523    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:56:03.808602 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:56:03.808518    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:56:03.886250 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:56:03.886212    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:56:03.957837 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:56:03.957807    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:56:04.058374 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:56:04.058339    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:56:04.159299 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:56:04.159268    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:56:04.362514 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:56:04.362483    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:56:04.474677 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:56:04.474587    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:56:04.564794 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:56:04.564759    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:56:45.539258 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:56:45.539222    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:56:46.918750 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:56:46.918702    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:56:51.790148 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:56:51.790088    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:56:51.845101 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:56:51.845071    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:56:56.842708 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:56:56.842675    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:56:57.561338 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:56:57.561298    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:00.461405 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:00.461367    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:00.562511 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:00.562478    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:02.157790 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:02.157751    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:02.275113 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:02.275081    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:02.399985 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:02.399954    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ltc2f"]
Jul 29 22:57:02.403169 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:02.403154    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-ltc2f"
Jul 29 22:57:02.405990 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:02.405972    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 29 22:57:02.406563 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:02.406547    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 29 22:57:02.421631 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:02.421607    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ltc2f"]
Jul 29 22:57:02.532646 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:02.532618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db9d0131-cfdc-46af-b8f8-b66b425635e8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ltc2f\" (UID: \"db9d0131-cfdc-46af-b8f8-b66b425635e8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ltc2f"
Jul 29 22:57:02.532794 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:02.532664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gkr9\" (UniqueName: \"kubernetes.io/projected/db9d0131-cfdc-46af-b8f8-b66b425635e8-kube-api-access-9gkr9\") pod \"cert-manager-webhook-6486ffd997-ltc2f\" (UID: \"db9d0131-cfdc-46af-b8f8-b66b425635e8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ltc2f"
Jul 29 22:57:02.633867 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:02.633838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db9d0131-cfdc-46af-b8f8-b66b425635e8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ltc2f\" (UID: \"db9d0131-cfdc-46af-b8f8-b66b425635e8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ltc2f"
Jul 29 22:57:02.634024 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:02.633878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gkr9\" (UniqueName: \"kubernetes.io/projected/db9d0131-cfdc-46af-b8f8-b66b425635e8-kube-api-access-9gkr9\") pod \"cert-manager-webhook-6486ffd997-ltc2f\" (UID: \"db9d0131-cfdc-46af-b8f8-b66b425635e8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ltc2f"
Jul 29 22:57:02.665982 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:02.665921    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db9d0131-cfdc-46af-b8f8-b66b425635e8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ltc2f\" (UID: \"db9d0131-cfdc-46af-b8f8-b66b425635e8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ltc2f"
Jul 29 22:57:02.669082 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:02.669061    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gkr9\" (UniqueName: \"kubernetes.io/projected/db9d0131-cfdc-46af-b8f8-b66b425635e8-kube-api-access-9gkr9\") pod \"cert-manager-webhook-6486ffd997-ltc2f\" (UID: \"db9d0131-cfdc-46af-b8f8-b66b425635e8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ltc2f"
Jul 29 22:57:02.725793 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:02.725768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-ltc2f"
Jul 29 22:57:02.858979 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:02.858870    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ltc2f"]
Jul 29 22:57:02.861630 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:57:02.861605    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb9d0131_cfdc_46af_b8f8_b66b425635e8.slice/crio-014040e39ccd43fb759523444f64c55381f6616d2eb43b15ef8ecbb2a7522173 WatchSource:0}: Error finding container 014040e39ccd43fb759523444f64c55381f6616d2eb43b15ef8ecbb2a7522173: Status 404 returned error can't find the container with id 014040e39ccd43fb759523444f64c55381f6616d2eb43b15ef8ecbb2a7522173
Jul 29 22:57:02.863540 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:02.863524    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 22:57:03.061302 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:03.061222    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:03.179196 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:03.179159    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:03.204023 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:03.203991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-ltc2f" event={"ID":"db9d0131-cfdc-46af-b8f8-b66b425635e8","Type":"ContainerStarted","Data":"014040e39ccd43fb759523444f64c55381f6616d2eb43b15ef8ecbb2a7522173"}
Jul 29 22:57:04.574907 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:04.574864    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:04.899972 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:04.899931    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:05.088094 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:05.088057    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:05.276012 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:05.275902    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:05.378779 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:05.378732    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:05.911312 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:05.911281    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-hzggd"]
Jul 29 22:57:05.914688 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:05.914666    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hzggd"
Jul 29 22:57:05.917790 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:05.917768    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-k8wm9\""
Jul 29 22:57:05.924683 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:05.924661    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-hzggd"]
Jul 29 22:57:05.964176 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:05.964153    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v8dt\" (UniqueName: \"kubernetes.io/projected/625c8160-241a-44b3-83aa-9728c1861a4a-kube-api-access-2v8dt\") pod \"cert-manager-cainjector-559cdd8c87-hzggd\" (UID: \"625c8160-241a-44b3-83aa-9728c1861a4a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hzggd"
Jul 29 22:57:05.964315 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:05.964238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/625c8160-241a-44b3-83aa-9728c1861a4a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-hzggd\" (UID: \"625c8160-241a-44b3-83aa-9728c1861a4a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hzggd"
Jul 29 22:57:06.065010 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:06.064956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v8dt\" (UniqueName: \"kubernetes.io/projected/625c8160-241a-44b3-83aa-9728c1861a4a-kube-api-access-2v8dt\") pod \"cert-manager-cainjector-559cdd8c87-hzggd\" (UID: \"625c8160-241a-44b3-83aa-9728c1861a4a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hzggd"
Jul 29 22:57:06.065169 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:06.065056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/625c8160-241a-44b3-83aa-9728c1861a4a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-hzggd\" (UID: \"625c8160-241a-44b3-83aa-9728c1861a4a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hzggd"
Jul 29 22:57:06.075763 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:06.075730    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/625c8160-241a-44b3-83aa-9728c1861a4a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-hzggd\" (UID: \"625c8160-241a-44b3-83aa-9728c1861a4a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hzggd"
Jul 29 22:57:06.075910 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:06.075888    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v8dt\" (UniqueName: \"kubernetes.io/projected/625c8160-241a-44b3-83aa-9728c1861a4a-kube-api-access-2v8dt\") pod \"cert-manager-cainjector-559cdd8c87-hzggd\" (UID: \"625c8160-241a-44b3-83aa-9728c1861a4a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hzggd"
Jul 29 22:57:06.226265 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:06.226174    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hzggd"
Jul 29 22:57:06.886206 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:06.886179    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-hzggd"]
Jul 29 22:57:06.888203 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:57:06.888169    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod625c8160_241a_44b3_83aa_9728c1861a4a.slice/crio-54983e13554b4f2da7753a3fc502dbb8e4bc8c9258f2d75e5f6a18e20e70577f WatchSource:0}: Error finding container 54983e13554b4f2da7753a3fc502dbb8e4bc8c9258f2d75e5f6a18e20e70577f: Status 404 returned error can't find the container with id 54983e13554b4f2da7753a3fc502dbb8e4bc8c9258f2d75e5f6a18e20e70577f
Jul 29 22:57:06.969291 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:06.969261    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:07.070474 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:07.070402    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:07.141849 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:07.141812    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:07.224944 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:07.224899    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hzggd" event={"ID":"625c8160-241a-44b3-83aa-9728c1861a4a","Type":"ContainerStarted","Data":"02bf67ec86d296da26c2357b2c789883c73035c5046dab8625a0223b0fbfe029"}
Jul 29 22:57:07.224944 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:07.224947    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hzggd" event={"ID":"625c8160-241a-44b3-83aa-9728c1861a4a","Type":"ContainerStarted","Data":"54983e13554b4f2da7753a3fc502dbb8e4bc8c9258f2d75e5f6a18e20e70577f"}
Jul 29 22:57:07.226484 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:07.226459    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-ltc2f" event={"ID":"db9d0131-cfdc-46af-b8f8-b66b425635e8","Type":"ContainerStarted","Data":"e38e9e07d774fca8960ae02a4c164b1ffe59ef496738807f9494fdc4b889ca64"}
Jul 29 22:57:07.226605 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:07.226533    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-ltc2f"
Jul 29 22:57:07.252993 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:07.252947    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hzggd" podStartSLOduration=2.252934157 podStartE2EDuration="2.252934157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:57:07.251899078 +0000 UTC m=+649.810243894" watchObservedRunningTime="2026-07-29 22:57:07.252934157 +0000 UTC m=+649.811278973"
Jul 29 22:57:07.290736 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:07.290687    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-ltc2f" podStartSLOduration=1.326885057 podStartE2EDuration="5.290671255s" totalImagesPullingTime="3.963786198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:57:07.289894275 +0000 UTC m=+649.848239092" watchObservedRunningTime="2026-07-29 22:57:07.290671255 +0000 UTC m=+649.849016070"
Jul 29 22:57:12.356230 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:12.356198    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:13.232287 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:13.232257    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-ltc2f"
Jul 29 22:57:16.769386 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:16.769349    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:18.350347 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:18.350301    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:18.449392 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:18.449357    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:20.178210 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:20.178181    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm88t"]
Jul 29 22:57:20.184829 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:20.184813    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm88t"
Jul 29 22:57:20.187735 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:20.187710    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 22:57:20.189181 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:20.189142    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 22:57:20.191726 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:20.189707    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-jq7vf\""
Jul 29 22:57:20.191726 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:20.190631    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm88t"]
Jul 29 22:57:20.273335 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:20.273302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8xww\" (UniqueName: \"kubernetes.io/projected/f268088a-1c18-42e9-892a-b9b0e9e860a7-kube-api-access-r8xww\") pod \"openshift-lws-operator-fd8ccff4c-dm88t\" (UID: \"f268088a-1c18-42e9-892a-b9b0e9e860a7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm88t"
Jul 29 22:57:20.273438 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:20.273348    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f268088a-1c18-42e9-892a-b9b0e9e860a7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dm88t\" (UID: \"f268088a-1c18-42e9-892a-b9b0e9e860a7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm88t"
Jul 29 22:57:20.373772 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:20.373750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f268088a-1c18-42e9-892a-b9b0e9e860a7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dm88t\" (UID: \"f268088a-1c18-42e9-892a-b9b0e9e860a7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm88t"
Jul 29 22:57:20.373870 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:20.373816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8xww\" (UniqueName: \"kubernetes.io/projected/f268088a-1c18-42e9-892a-b9b0e9e860a7-kube-api-access-r8xww\") pod \"openshift-lws-operator-fd8ccff4c-dm88t\" (UID: \"f268088a-1c18-42e9-892a-b9b0e9e860a7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm88t"
Jul 29 22:57:20.374121 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:20.374103    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f268088a-1c18-42e9-892a-b9b0e9e860a7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dm88t\" (UID: \"f268088a-1c18-42e9-892a-b9b0e9e860a7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm88t"
Jul 29 22:57:20.382254 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:20.382232    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8xww\" (UniqueName: \"kubernetes.io/projected/f268088a-1c18-42e9-892a-b9b0e9e860a7-kube-api-access-r8xww\") pod \"openshift-lws-operator-fd8ccff4c-dm88t\" (UID: \"f268088a-1c18-42e9-892a-b9b0e9e860a7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm88t"
Jul 29 22:57:20.495332 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:20.495283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm88t"
Jul 29 22:57:20.625553 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:20.625528    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm88t"]
Jul 29 22:57:20.628039 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:57:20.628013    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf268088a_1c18_42e9_892a_b9b0e9e860a7.slice/crio-d8136a515ebd45b4f2a139b349823b27687147929d0d9668a18418b45bfa3674 WatchSource:0}: Error finding container d8136a515ebd45b4f2a139b349823b27687147929d0d9668a18418b45bfa3674: Status 404 returned error can't find the container with id d8136a515ebd45b4f2a139b349823b27687147929d0d9668a18418b45bfa3674
Jul 29 22:57:21.283005 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:21.282971    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm88t" event={"ID":"f268088a-1c18-42e9-892a-b9b0e9e860a7","Type":"ContainerStarted","Data":"d8136a515ebd45b4f2a139b349823b27687147929d0d9668a18418b45bfa3674"}
Jul 29 22:57:24.296641 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:24.296609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm88t" event={"ID":"f268088a-1c18-42e9-892a-b9b0e9e860a7","Type":"ContainerStarted","Data":"d68c35882e70e534689435c504145f5307f6ce296f499d9db984ef466c87fc4f"}
Jul 29 22:57:24.314162 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:24.314118    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm88t" podStartSLOduration=1.6409443000000001 podStartE2EDuration="4.314103231s" totalImagesPullingTime="2.673158931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:57:24.313158562 +0000 UTC m=+666.871503378" watchObservedRunningTime="2026-07-29 22:57:24.314103231 +0000 UTC m=+666.872448047"
Jul 29 22:57:26.092961 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:26.092906    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:26.191127 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:26.191101    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:30.641700 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:30.641665    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:35.617601 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:35.617567    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:35.684219 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:35.684187    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:36.591939 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:36.591902    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:36.872088 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:36.872006    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:36.903538 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:36.903504    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"]
Jul 29 22:57:36.906852 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:36.906834    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"
Jul 29 22:57:36.910988 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:36.910965    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 29 22:57:36.910988 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:36.910980    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 29 22:57:36.911166 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:36.911040    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 29 22:57:36.911166 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:36.910983    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-v66rt\""
Jul 29 22:57:36.916298 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:36.916279    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"]
Jul 29 22:57:36.978232 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:36.978198    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:37.006986 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:37.006960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da621695-b3b9-4d0e-b772-b0a1c4b3a15c-manager-config\") pod \"lws-controller-manager-78dcd69c74-c9ztf\" (UID: \"da621695-b3b9-4d0e-b772-b0a1c4b3a15c\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"
Jul 29 22:57:37.007110 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:37.006988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/da621695-b3b9-4d0e-b772-b0a1c4b3a15c-metrics-cert\") pod \"lws-controller-manager-78dcd69c74-c9ztf\" (UID: \"da621695-b3b9-4d0e-b772-b0a1c4b3a15c\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"
Jul 29 22:57:37.007110 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:37.007024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtspg\" (UniqueName: \"kubernetes.io/projected/da621695-b3b9-4d0e-b772-b0a1c4b3a15c-kube-api-access-gtspg\") pod \"lws-controller-manager-78dcd69c74-c9ztf\" (UID: \"da621695-b3b9-4d0e-b772-b0a1c4b3a15c\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"
Jul 29 22:57:37.007200 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:37.007134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da621695-b3b9-4d0e-b772-b0a1c4b3a15c-cert\") pod \"lws-controller-manager-78dcd69c74-c9ztf\" (UID: \"da621695-b3b9-4d0e-b772-b0a1c4b3a15c\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"
Jul 29 22:57:37.108266 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:37.108233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da621695-b3b9-4d0e-b772-b0a1c4b3a15c-manager-config\") pod \"lws-controller-manager-78dcd69c74-c9ztf\" (UID: \"da621695-b3b9-4d0e-b772-b0a1c4b3a15c\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"
Jul 29 22:57:37.108266 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:37.108267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/da621695-b3b9-4d0e-b772-b0a1c4b3a15c-metrics-cert\") pod \"lws-controller-manager-78dcd69c74-c9ztf\" (UID: \"da621695-b3b9-4d0e-b772-b0a1c4b3a15c\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"
Jul 29 22:57:37.108455 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:37.108294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtspg\" (UniqueName: \"kubernetes.io/projected/da621695-b3b9-4d0e-b772-b0a1c4b3a15c-kube-api-access-gtspg\") pod \"lws-controller-manager-78dcd69c74-c9ztf\" (UID: \"da621695-b3b9-4d0e-b772-b0a1c4b3a15c\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"
Jul 29 22:57:37.108455 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:37.108329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da621695-b3b9-4d0e-b772-b0a1c4b3a15c-cert\") pod \"lws-controller-manager-78dcd69c74-c9ztf\" (UID: \"da621695-b3b9-4d0e-b772-b0a1c4b3a15c\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"
Jul 29 22:57:37.108986 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:37.108960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da621695-b3b9-4d0e-b772-b0a1c4b3a15c-manager-config\") pod \"lws-controller-manager-78dcd69c74-c9ztf\" (UID: \"da621695-b3b9-4d0e-b772-b0a1c4b3a15c\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"
Jul 29 22:57:37.110742 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:37.110724    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da621695-b3b9-4d0e-b772-b0a1c4b3a15c-cert\") pod \"lws-controller-manager-78dcd69c74-c9ztf\" (UID: \"da621695-b3b9-4d0e-b772-b0a1c4b3a15c\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"
Jul 29 22:57:37.110828 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:37.110788    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/da621695-b3b9-4d0e-b772-b0a1c4b3a15c-metrics-cert\") pod \"lws-controller-manager-78dcd69c74-c9ztf\" (UID: \"da621695-b3b9-4d0e-b772-b0a1c4b3a15c\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"
Jul 29 22:57:37.118546 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:37.118524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtspg\" (UniqueName: \"kubernetes.io/projected/da621695-b3b9-4d0e-b772-b0a1c4b3a15c-kube-api-access-gtspg\") pod \"lws-controller-manager-78dcd69c74-c9ztf\" (UID: \"da621695-b3b9-4d0e-b772-b0a1c4b3a15c\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"
Jul 29 22:57:37.216698 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:37.216622    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"
Jul 29 22:57:37.362504 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:37.362483    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"]
Jul 29 22:57:37.364402 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:57:37.364380    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda621695_b3b9_4d0e_b772_b0a1c4b3a15c.slice/crio-89607be42b7a6ea8f63b7da7cb80f5654f5dc368d00a616e84dcbf39359e9077 WatchSource:0}: Error finding container 89607be42b7a6ea8f63b7da7cb80f5654f5dc368d00a616e84dcbf39359e9077: Status 404 returned error can't find the container with id 89607be42b7a6ea8f63b7da7cb80f5654f5dc368d00a616e84dcbf39359e9077
Jul 29 22:57:38.354507 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:38.354472    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf" event={"ID":"da621695-b3b9-4d0e-b772-b0a1c4b3a15c","Type":"ContainerStarted","Data":"89607be42b7a6ea8f63b7da7cb80f5654f5dc368d00a616e84dcbf39359e9077"}
Jul 29 22:57:39.360421 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:39.360381    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf" event={"ID":"da621695-b3b9-4d0e-b772-b0a1c4b3a15c","Type":"ContainerStarted","Data":"334ad8285d63068ac418c861da6120715e51bff43696911eab87b40c33ee85ee"}
Jul 29 22:57:39.360718 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:39.360479    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"
Jul 29 22:57:39.380891 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:39.380834    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf" podStartSLOduration=1.453350425 podStartE2EDuration="3.380811866s" totalImagesPullingTime="1.927461441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:57:39.378342988 +0000 UTC m=+681.936687806" watchObservedRunningTime="2026-07-29 22:57:39.380811866 +0000 UTC m=+681.939156675"
Jul 29 22:57:40.064784 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:40.064746    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:40.167266 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:40.167225    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:42.069792 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:42.069752    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:42.151028 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:42.150987    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:45.683957 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:45.683927    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:47.554523 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:47.554487    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:47.651166 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:47.651134    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:57:50.366142 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:57:50.366113    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-c9ztf"
Jul 29 22:58:01.081191 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:01.081147    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:02.099866 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:02.099829    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:06.896422 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:06.896390    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:07.373509 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:07.373481    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:07.476457 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:07.476428    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:09.321775 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:09.321741    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-48g2h"]
Jul 29 22:58:09.326069 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:09.326049    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-48g2h"
Jul 29 22:58:09.330296 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:09.330276    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 22:58:09.330412 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:09.330335    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 22:58:09.331511 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:09.331488    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5557j\""
Jul 29 22:58:09.339012 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:09.338991    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-48g2h"]
Jul 29 22:58:09.373315 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:09.373294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9vr8\" (UniqueName: \"kubernetes.io/projected/30e566be-2326-4a4f-bf39-ef84cd8aaa97-kube-api-access-g9vr8\") pod \"limitador-operator-controller-manager-774d94476f-48g2h\" (UID: \"30e566be-2326-4a4f-bf39-ef84cd8aaa97\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-48g2h"
Jul 29 22:58:09.474239 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:09.474177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9vr8\" (UniqueName: \"kubernetes.io/projected/30e566be-2326-4a4f-bf39-ef84cd8aaa97-kube-api-access-g9vr8\") pod \"limitador-operator-controller-manager-774d94476f-48g2h\" (UID: \"30e566be-2326-4a4f-bf39-ef84cd8aaa97\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-48g2h"
Jul 29 22:58:09.486187 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:09.486164    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9vr8\" (UniqueName: \"kubernetes.io/projected/30e566be-2326-4a4f-bf39-ef84cd8aaa97-kube-api-access-g9vr8\") pod \"limitador-operator-controller-manager-774d94476f-48g2h\" (UID: \"30e566be-2326-4a4f-bf39-ef84cd8aaa97\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-48g2h"
Jul 29 22:58:09.635808 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:09.635781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-48g2h"
Jul 29 22:58:09.765387 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:09.765360    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-48g2h"]
Jul 29 22:58:09.768082 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:58:09.768056    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e566be_2326_4a4f_bf39_ef84cd8aaa97.slice/crio-c8efd8cfb5a4131903e757d79f0a5e58eeead25e444eedbb1fe7a9cb0c4f33e8 WatchSource:0}: Error finding container c8efd8cfb5a4131903e757d79f0a5e58eeead25e444eedbb1fe7a9cb0c4f33e8: Status 404 returned error can't find the container with id c8efd8cfb5a4131903e757d79f0a5e58eeead25e444eedbb1fe7a9cb0c4f33e8
Jul 29 22:58:10.334484 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:10.334448    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:10.495749 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:10.495713    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-48g2h" event={"ID":"30e566be-2326-4a4f-bf39-ef84cd8aaa97","Type":"ContainerStarted","Data":"c8efd8cfb5a4131903e757d79f0a5e58eeead25e444eedbb1fe7a9cb0c4f33e8"}
Jul 29 22:58:10.740928 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:10.740843    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:13.297944 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:13.297902    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:13.511675 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:13.511643    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-48g2h" event={"ID":"30e566be-2326-4a4f-bf39-ef84cd8aaa97","Type":"ContainerStarted","Data":"d17e320920b77f9b17f2ab28792434596f866fcef77c7d38962fdd9872961116"}
Jul 29 22:58:13.511859 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:13.511731    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-48g2h"
Jul 29 22:58:13.531070 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:13.531025    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-48g2h" podStartSLOduration=1.7788372510000001 podStartE2EDuration="4.53101018s" totalImagesPullingTime="2.752172929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:58:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:58:13.529200701 +0000 UTC m=+716.087545514" watchObservedRunningTime="2026-07-29 22:58:13.53101018 +0000 UTC m=+716.089354996"
Jul 29 22:58:13.886242 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:13.886207    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:14.679004 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:14.678969    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:14.954477 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:14.954397    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:16.941193 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:16.941158    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:20.913384 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:20.913346    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:21.017814 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:21.017770    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:22.084839 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:22.084809    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:22.185045 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:22.185012    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:24.518184 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:24.518156    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-48g2h"
Jul 29 22:58:25.761403 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:25.761365    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:25.898237 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:25.898201    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:25.998821 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:25.998763    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:26.165479 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:26.165438    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:28.484381 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:28.484343    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:28.584444 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:28.584410    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:30.123633 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:30.123597    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:30.183185 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:30.183152    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:33.014117 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:33.014081    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:34.445286 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:34.445253    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:34.543857 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:34.543823    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:34.628769 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:34.628739    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:42.311258 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:42.311225    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:42.374105 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:42.374079    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:42.916029 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:42.915992    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:43.018140 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:43.018112    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:46.310890 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:46.310850    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:46.414528 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:46.414491    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:48.707736 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:48.707695    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:48.765733 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:48.765697    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:54.472511 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:54.472470    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:54.572854 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:54.572819    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:58:54.635130 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:58:54.635090    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:59:10.313320 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:10.313283    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"]
Jul 29 22:59:10.314510 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:10.314480    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:59:10.318056 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:10.318041    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"
Jul 29 22:59:10.321001 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:10.320981    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 29 22:59:10.321092 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:10.321021    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m864v\""
Jul 29 22:59:10.324352 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:10.324331    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"]
Jul 29 22:59:10.402098 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:10.402066    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"]
Jul 29 22:59:10.413146 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:10.413102    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:59:10.429684 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:10.429659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmvr\" (UniqueName: \"kubernetes.io/projected/db61b5b9-3772-41c1-896f-c08db21c3fb7-kube-api-access-5tmvr\") pod \"limitador-limitador-859c4d96d6-2sw6r\" (UID: \"db61b5b9-3772-41c1-896f-c08db21c3fb7\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"
Jul 29 22:59:10.429792 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:10.429737    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/db61b5b9-3772-41c1-896f-c08db21c3fb7-config-file\") pod \"limitador-limitador-859c4d96d6-2sw6r\" (UID: \"db61b5b9-3772-41c1-896f-c08db21c3fb7\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"
Jul 29 22:59:10.479514 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:10.479490    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:59:10.530651 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:10.530621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/db61b5b9-3772-41c1-896f-c08db21c3fb7-config-file\") pod \"limitador-limitador-859c4d96d6-2sw6r\" (UID: \"db61b5b9-3772-41c1-896f-c08db21c3fb7\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"
Jul 29 22:59:10.530788 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:10.530687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmvr\" (UniqueName: \"kubernetes.io/projected/db61b5b9-3772-41c1-896f-c08db21c3fb7-kube-api-access-5tmvr\") pod \"limitador-limitador-859c4d96d6-2sw6r\" (UID: \"db61b5b9-3772-41c1-896f-c08db21c3fb7\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"
Jul 29 22:59:10.531268 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:10.531250    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/db61b5b9-3772-41c1-896f-c08db21c3fb7-config-file\") pod \"limitador-limitador-859c4d96d6-2sw6r\" (UID: \"db61b5b9-3772-41c1-896f-c08db21c3fb7\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"
Jul 29 22:59:10.538356 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:10.538334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmvr\" (UniqueName: \"kubernetes.io/projected/db61b5b9-3772-41c1-896f-c08db21c3fb7-kube-api-access-5tmvr\") pod \"limitador-limitador-859c4d96d6-2sw6r\" (UID: \"db61b5b9-3772-41c1-896f-c08db21c3fb7\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"
Jul 29 22:59:10.628285 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:10.628256    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"
Jul 29 22:59:10.762537 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:10.762513    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"]
Jul 29 22:59:10.765366 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:59:10.765344    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb61b5b9_3772_41c1_896f_c08db21c3fb7.slice/crio-7ffaea99858964ec74cd85d5a7eb45fae1e7d063d8b2cc13afc0eece31c7fbf3 WatchSource:0}: Error finding container 7ffaea99858964ec74cd85d5a7eb45fae1e7d063d8b2cc13afc0eece31c7fbf3: Status 404 returned error can't find the container with id 7ffaea99858964ec74cd85d5a7eb45fae1e7d063d8b2cc13afc0eece31c7fbf3
Jul 29 22:59:10.772716 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:10.772685    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-2sw6r" event={"ID":"db61b5b9-3772-41c1-896f-c08db21c3fb7","Type":"ContainerStarted","Data":"7ffaea99858964ec74cd85d5a7eb45fae1e7d063d8b2cc13afc0eece31c7fbf3"}
Jul 29 22:59:11.144449 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:11.144415    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:59:11.243640 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:11.243613    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:59:13.812469 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:13.811596    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-2sw6r" event={"ID":"db61b5b9-3772-41c1-896f-c08db21c3fb7","Type":"ContainerStarted","Data":"86111777baf1e0231850124a36283a8e0407efd45f82d8544a4f2d7b1a3f3f1b"}
Jul 29 22:59:13.812469 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:13.812407    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"
Jul 29 22:59:13.831711 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:13.831646    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-2sw6r" podStartSLOduration=1.274153834 podStartE2EDuration="3.831625925s" totalImagesPullingTime="2.557472091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:59:13.829292955 +0000 UTC m=+776.387637791" watchObservedRunningTime="2026-07-29 22:59:13.831625925 +0000 UTC m=+776.389970742"
Jul 29 22:59:23.631705 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:23.631670    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"]
Jul 29 22:59:23.632250 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:23.631890    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-2sw6r" podUID="db61b5b9-3772-41c1-896f-c08db21c3fb7" containerName="limitador" containerID="cri-o://86111777baf1e0231850124a36283a8e0407efd45f82d8544a4f2d7b1a3f3f1b" gracePeriod=30
Jul 29 22:59:23.632646 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:23.632441    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"
Jul 29 22:59:24.175540 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:24.175518    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"
Jul 29 22:59:24.241126 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:24.241056    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tmvr\" (UniqueName: \"kubernetes.io/projected/db61b5b9-3772-41c1-896f-c08db21c3fb7-kube-api-access-5tmvr\") pod \"db61b5b9-3772-41c1-896f-c08db21c3fb7\" (UID: \"db61b5b9-3772-41c1-896f-c08db21c3fb7\") "
Jul 29 22:59:24.241274 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:24.241140    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/db61b5b9-3772-41c1-896f-c08db21c3fb7-config-file\") pod \"db61b5b9-3772-41c1-896f-c08db21c3fb7\" (UID: \"db61b5b9-3772-41c1-896f-c08db21c3fb7\") "
Jul 29 22:59:24.241455 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:24.241430    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db61b5b9-3772-41c1-896f-c08db21c3fb7-config-file" (OuterVolumeSpecName: "config-file") pod "db61b5b9-3772-41c1-896f-c08db21c3fb7" (UID: "db61b5b9-3772-41c1-896f-c08db21c3fb7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:59:24.243142 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:24.243122    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db61b5b9-3772-41c1-896f-c08db21c3fb7-kube-api-access-5tmvr" (OuterVolumeSpecName: "kube-api-access-5tmvr") pod "db61b5b9-3772-41c1-896f-c08db21c3fb7" (UID: "db61b5b9-3772-41c1-896f-c08db21c3fb7"). InnerVolumeSpecName "kube-api-access-5tmvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:59:24.342234 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:24.342200    2581 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/db61b5b9-3772-41c1-896f-c08db21c3fb7-config-file\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:59:24.342234 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:24.342231    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tmvr\" (UniqueName: \"kubernetes.io/projected/db61b5b9-3772-41c1-896f-c08db21c3fb7-kube-api-access-5tmvr\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 22:59:24.862424 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:24.862390    2581 generic.go:356] "Generic (PLEG): container finished" podID="db61b5b9-3772-41c1-896f-c08db21c3fb7" containerID="86111777baf1e0231850124a36283a8e0407efd45f82d8544a4f2d7b1a3f3f1b" exitCode=0
Jul 29 22:59:24.862870 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:24.862441    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"
Jul 29 22:59:24.862870 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:24.862437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-2sw6r" event={"ID":"db61b5b9-3772-41c1-896f-c08db21c3fb7","Type":"ContainerDied","Data":"86111777baf1e0231850124a36283a8e0407efd45f82d8544a4f2d7b1a3f3f1b"}
Jul 29 22:59:24.862870 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:24.862588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-2sw6r" event={"ID":"db61b5b9-3772-41c1-896f-c08db21c3fb7","Type":"ContainerDied","Data":"7ffaea99858964ec74cd85d5a7eb45fae1e7d063d8b2cc13afc0eece31c7fbf3"}
Jul 29 22:59:24.862870 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:24.862612    2581 scope.go:117] "RemoveContainer" containerID="86111777baf1e0231850124a36283a8e0407efd45f82d8544a4f2d7b1a3f3f1b"
Jul 29 22:59:24.871668 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:24.871653    2581 scope.go:117] "RemoveContainer" containerID="86111777baf1e0231850124a36283a8e0407efd45f82d8544a4f2d7b1a3f3f1b"
Jul 29 22:59:24.871933 ip-10-0-128-242 kubenswrapper[2581]: E0729 22:59:24.871915    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86111777baf1e0231850124a36283a8e0407efd45f82d8544a4f2d7b1a3f3f1b\": container with ID starting with 86111777baf1e0231850124a36283a8e0407efd45f82d8544a4f2d7b1a3f3f1b not found: ID does not exist" containerID="86111777baf1e0231850124a36283a8e0407efd45f82d8544a4f2d7b1a3f3f1b"
Jul 29 22:59:24.872008 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:24.871943    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86111777baf1e0231850124a36283a8e0407efd45f82d8544a4f2d7b1a3f3f1b"} err="failed to get container status \"86111777baf1e0231850124a36283a8e0407efd45f82d8544a4f2d7b1a3f3f1b\": rpc error: code = NotFound desc = could not find container \"86111777baf1e0231850124a36283a8e0407efd45f82d8544a4f2d7b1a3f3f1b\": container with ID starting with 86111777baf1e0231850124a36283a8e0407efd45f82d8544a4f2d7b1a3f3f1b not found: ID does not exist"
Jul 29 22:59:24.887984 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:24.887958    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"]
Jul 29 22:59:24.896078 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:24.896058    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2sw6r"]
Jul 29 22:59:25.951853 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:25.951817    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db61b5b9-3772-41c1-896f-c08db21c3fb7" path="/var/lib/kubelet/pods/db61b5b9-3772-41c1-896f-c08db21c3fb7/volumes"
Jul 29 22:59:29.654662 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:29.654618    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:59:30.072319 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:30.072225    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:59:33.553457 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:33.553420    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cf4765b85-gvscv"]
Jul 29 22:59:33.553962 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:33.553759    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db61b5b9-3772-41c1-896f-c08db21c3fb7" containerName="limitador"
Jul 29 22:59:33.553962 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:33.553772    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="db61b5b9-3772-41c1-896f-c08db21c3fb7" containerName="limitador"
Jul 29 22:59:33.553962 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:33.553872    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="db61b5b9-3772-41c1-896f-c08db21c3fb7" containerName="limitador"
Jul 29 22:59:33.558283 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:33.558264    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-gvscv"
Jul 29 22:59:33.562421 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:33.562403    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 29 22:59:33.562532 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:33.562422    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8w8rg\""
Jul 29 22:59:33.564540 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:33.564514    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-gvscv"]
Jul 29 22:59:33.609867 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:33.609846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hr29\" (UniqueName: \"kubernetes.io/projected/78f0b0f8-0335-49a3-88f4-d47d5801d011-kube-api-access-5hr29\") pod \"authorino-7cf4765b85-gvscv\" (UID: \"78f0b0f8-0335-49a3-88f4-d47d5801d011\") " pod="kuadrant-system/authorino-7cf4765b85-gvscv"
Jul 29 22:59:33.609970 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:33.609906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/78f0b0f8-0335-49a3-88f4-d47d5801d011-tls-cert\") pod \"authorino-7cf4765b85-gvscv\" (UID: \"78f0b0f8-0335-49a3-88f4-d47d5801d011\") " pod="kuadrant-system/authorino-7cf4765b85-gvscv"
Jul 29 22:59:33.711219 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:33.711183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hr29\" (UniqueName: \"kubernetes.io/projected/78f0b0f8-0335-49a3-88f4-d47d5801d011-kube-api-access-5hr29\") pod \"authorino-7cf4765b85-gvscv\" (UID: \"78f0b0f8-0335-49a3-88f4-d47d5801d011\") " pod="kuadrant-system/authorino-7cf4765b85-gvscv"
Jul 29 22:59:33.711321 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:33.711278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/78f0b0f8-0335-49a3-88f4-d47d5801d011-tls-cert\") pod \"authorino-7cf4765b85-gvscv\" (UID: \"78f0b0f8-0335-49a3-88f4-d47d5801d011\") " pod="kuadrant-system/authorino-7cf4765b85-gvscv"
Jul 29 22:59:33.713704 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:33.713676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/78f0b0f8-0335-49a3-88f4-d47d5801d011-tls-cert\") pod \"authorino-7cf4765b85-gvscv\" (UID: \"78f0b0f8-0335-49a3-88f4-d47d5801d011\") " pod="kuadrant-system/authorino-7cf4765b85-gvscv"
Jul 29 22:59:33.719789 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:33.719761    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hr29\" (UniqueName: \"kubernetes.io/projected/78f0b0f8-0335-49a3-88f4-d47d5801d011-kube-api-access-5hr29\") pod \"authorino-7cf4765b85-gvscv\" (UID: \"78f0b0f8-0335-49a3-88f4-d47d5801d011\") " pod="kuadrant-system/authorino-7cf4765b85-gvscv"
Jul 29 22:59:33.867564 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:33.867545    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-gvscv"
Jul 29 22:59:33.996009 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:33.995978    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-gvscv"]
Jul 29 22:59:33.998831 ip-10-0-128-242 kubenswrapper[2581]: W0729 22:59:33.998789    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78f0b0f8_0335_49a3_88f4_d47d5801d011.slice/crio-1b7a3d29095071c68e2183cb7b809622540ecbb11d32d41b08f76744f41f4d38 WatchSource:0}: Error finding container 1b7a3d29095071c68e2183cb7b809622540ecbb11d32d41b08f76744f41f4d38: Status 404 returned error can't find the container with id 1b7a3d29095071c68e2183cb7b809622540ecbb11d32d41b08f76744f41f4d38
Jul 29 22:59:34.911034 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:34.910987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-gvscv" event={"ID":"78f0b0f8-0335-49a3-88f4-d47d5801d011","Type":"ContainerStarted","Data":"1b7a3d29095071c68e2183cb7b809622540ecbb11d32d41b08f76744f41f4d38"}
Jul 29 22:59:35.917746 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:35.917706    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-gvscv" event={"ID":"78f0b0f8-0335-49a3-88f4-d47d5801d011","Type":"ContainerStarted","Data":"1d65cf5fb4cd531bb9c974e6d6a0a6d9e25ae1e4d25eb9922dd95aceed8eff78"}
Jul 29 22:59:35.933775 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:35.933730    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cf4765b85-gvscv" podStartSLOduration=1.22329815 podStartE2EDuration="2.933715338s" totalImagesPullingTime="1.710417188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:59:35.933420547 +0000 UTC m=+798.491765362" watchObservedRunningTime="2026-07-29 22:59:35.933715338 +0000 UTC m=+798.492060155"
Jul 29 22:59:40.404910 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:40.404831    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 22:59:40.477449 ip-10-0-128-242 kubenswrapper[2581]: I0729 22:59:40.477420    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:00:18.705130 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.705091    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:00:18.730329 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.730301    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-6t2tn"]
Jul 29 23:00:18.733963 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.733942    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-6t2tn"
Jul 29 23:00:18.736918 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.736900    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 29 23:00:18.737024 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.736995    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-r79sr\""
Jul 29 23:00:18.737113 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.736943    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 29 23:00:18.738838 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.738821    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 29 23:00:18.740521 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.740501    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-6t2tn"]
Jul 29 23:00:18.745600 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.745579    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-8588677cfb-n4kbt"]
Jul 29 23:00:18.748375 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.748360    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-n4kbt"
Jul 29 23:00:18.751902 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.751686    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 29 23:00:18.751902 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.751764    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-sfdjh\""
Jul 29 23:00:18.752336 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.752175    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Jul 29 23:00:18.758203 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.758183    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-n4kbt"]
Jul 29 23:00:18.774988 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.774963    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:00:18.777187 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.777161    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e9778d8-4555-443c-91a5-eabd88c1fccb-cert\") pod \"odh-model-controller-8588677cfb-n4kbt\" (UID: \"5e9778d8-4555-443c-91a5-eabd88c1fccb\") " pod="kserve/odh-model-controller-8588677cfb-n4kbt"
Jul 29 23:00:18.777270 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.777213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-628x7\" (UniqueName: \"kubernetes.io/projected/5e9778d8-4555-443c-91a5-eabd88c1fccb-kube-api-access-628x7\") pod \"odh-model-controller-8588677cfb-n4kbt\" (UID: \"5e9778d8-4555-443c-91a5-eabd88c1fccb\") " pod="kserve/odh-model-controller-8588677cfb-n4kbt"
Jul 29 23:00:18.777324 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.777271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e42d3a61-ebda-44c1-9c22-deea87c1f921-tls-certs\") pod \"model-serving-api-7b954ffc95-6t2tn\" (UID: \"e42d3a61-ebda-44c1-9c22-deea87c1f921\") " pod="kserve/model-serving-api-7b954ffc95-6t2tn"
Jul 29 23:00:18.777324 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.777301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9778d8-4555-443c-91a5-eabd88c1fccb-metrics-certs\") pod \"odh-model-controller-8588677cfb-n4kbt\" (UID: \"5e9778d8-4555-443c-91a5-eabd88c1fccb\") " pod="kserve/odh-model-controller-8588677cfb-n4kbt"
Jul 29 23:00:18.777404 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.777338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8czrt\" (UniqueName: \"kubernetes.io/projected/e42d3a61-ebda-44c1-9c22-deea87c1f921-kube-api-access-8czrt\") pod \"model-serving-api-7b954ffc95-6t2tn\" (UID: \"e42d3a61-ebda-44c1-9c22-deea87c1f921\") " pod="kserve/model-serving-api-7b954ffc95-6t2tn"
Jul 29 23:00:18.877937 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.877900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e42d3a61-ebda-44c1-9c22-deea87c1f921-tls-certs\") pod \"model-serving-api-7b954ffc95-6t2tn\" (UID: \"e42d3a61-ebda-44c1-9c22-deea87c1f921\") " pod="kserve/model-serving-api-7b954ffc95-6t2tn"
Jul 29 23:00:18.877937 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.877938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9778d8-4555-443c-91a5-eabd88c1fccb-metrics-certs\") pod \"odh-model-controller-8588677cfb-n4kbt\" (UID: \"5e9778d8-4555-443c-91a5-eabd88c1fccb\") " pod="kserve/odh-model-controller-8588677cfb-n4kbt"
Jul 29 23:00:18.878183 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.877970    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czrt\" (UniqueName: \"kubernetes.io/projected/e42d3a61-ebda-44c1-9c22-deea87c1f921-kube-api-access-8czrt\") pod \"model-serving-api-7b954ffc95-6t2tn\" (UID: \"e42d3a61-ebda-44c1-9c22-deea87c1f921\") " pod="kserve/model-serving-api-7b954ffc95-6t2tn"
Jul 29 23:00:18.878183 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.878014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e9778d8-4555-443c-91a5-eabd88c1fccb-cert\") pod \"odh-model-controller-8588677cfb-n4kbt\" (UID: \"5e9778d8-4555-443c-91a5-eabd88c1fccb\") " pod="kserve/odh-model-controller-8588677cfb-n4kbt"
Jul 29 23:00:18.878183 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.878044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-628x7\" (UniqueName: \"kubernetes.io/projected/5e9778d8-4555-443c-91a5-eabd88c1fccb-kube-api-access-628x7\") pod \"odh-model-controller-8588677cfb-n4kbt\" (UID: \"5e9778d8-4555-443c-91a5-eabd88c1fccb\") " pod="kserve/odh-model-controller-8588677cfb-n4kbt"
Jul 29 23:00:18.878183 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:00:18.878051    2581 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 29 23:00:18.878183 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:00:18.878086    2581 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Jul 29 23:00:18.878183 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:00:18.878150    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e42d3a61-ebda-44c1-9c22-deea87c1f921-tls-certs podName:e42d3a61-ebda-44c1-9c22-deea87c1f921 nodeName:}" failed. No retries permitted until 2026-07-29 23:00:19.378113289 +0000 UTC m=+841.936458100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e42d3a61-ebda-44c1-9c22-deea87c1f921-tls-certs") pod "model-serving-api-7b954ffc95-6t2tn" (UID: "e42d3a61-ebda-44c1-9c22-deea87c1f921") : secret "model-serving-api-tls" not found
Jul 29 23:00:18.878183 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:00:18.878167    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e9778d8-4555-443c-91a5-eabd88c1fccb-metrics-certs podName:5e9778d8-4555-443c-91a5-eabd88c1fccb nodeName:}" failed. No retries permitted until 2026-07-29 23:00:19.378160384 +0000 UTC m=+841.936505178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e9778d8-4555-443c-91a5-eabd88c1fccb-metrics-certs") pod "odh-model-controller-8588677cfb-n4kbt" (UID: "5e9778d8-4555-443c-91a5-eabd88c1fccb") : secret "odh-model-controller-metrics-tls" not found
Jul 29 23:00:18.880479 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.880443    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e9778d8-4555-443c-91a5-eabd88c1fccb-cert\") pod \"odh-model-controller-8588677cfb-n4kbt\" (UID: \"5e9778d8-4555-443c-91a5-eabd88c1fccb\") " pod="kserve/odh-model-controller-8588677cfb-n4kbt"
Jul 29 23:00:18.886789 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.886767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-628x7\" (UniqueName: \"kubernetes.io/projected/5e9778d8-4555-443c-91a5-eabd88c1fccb-kube-api-access-628x7\") pod \"odh-model-controller-8588677cfb-n4kbt\" (UID: \"5e9778d8-4555-443c-91a5-eabd88c1fccb\") " pod="kserve/odh-model-controller-8588677cfb-n4kbt"
Jul 29 23:00:18.886908 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.886849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8czrt\" (UniqueName: \"kubernetes.io/projected/e42d3a61-ebda-44c1-9c22-deea87c1f921-kube-api-access-8czrt\") pod \"model-serving-api-7b954ffc95-6t2tn\" (UID: \"e42d3a61-ebda-44c1-9c22-deea87c1f921\") " pod="kserve/model-serving-api-7b954ffc95-6t2tn"
Jul 29 23:00:18.910544 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:18.910515    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:00:19.014888 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:19.014830    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:00:19.150561 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:19.150533    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:00:19.256362 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:19.256340    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:00:19.356669 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:19.356642    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:00:19.381090 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:19.381069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e42d3a61-ebda-44c1-9c22-deea87c1f921-tls-certs\") pod \"model-serving-api-7b954ffc95-6t2tn\" (UID: \"e42d3a61-ebda-44c1-9c22-deea87c1f921\") " pod="kserve/model-serving-api-7b954ffc95-6t2tn"
Jul 29 23:00:19.381172 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:19.381098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9778d8-4555-443c-91a5-eabd88c1fccb-metrics-certs\") pod \"odh-model-controller-8588677cfb-n4kbt\" (UID: \"5e9778d8-4555-443c-91a5-eabd88c1fccb\") " pod="kserve/odh-model-controller-8588677cfb-n4kbt"
Jul 29 23:00:19.383419 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:19.383394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e42d3a61-ebda-44c1-9c22-deea87c1f921-tls-certs\") pod \"model-serving-api-7b954ffc95-6t2tn\" (UID: \"e42d3a61-ebda-44c1-9c22-deea87c1f921\") " pod="kserve/model-serving-api-7b954ffc95-6t2tn"
Jul 29 23:00:19.383563 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:19.383543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9778d8-4555-443c-91a5-eabd88c1fccb-metrics-certs\") pod \"odh-model-controller-8588677cfb-n4kbt\" (UID: \"5e9778d8-4555-443c-91a5-eabd88c1fccb\") " pod="kserve/odh-model-controller-8588677cfb-n4kbt"
Jul 29 23:00:19.648093 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:19.648022    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-6t2tn"
Jul 29 23:00:19.661017 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:19.660988    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-n4kbt"
Jul 29 23:00:19.786383 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:19.786347    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-6t2tn"]
Jul 29 23:00:19.791750 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:00:19.791722    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42d3a61_ebda_44c1_9c22_deea87c1f921.slice/crio-215e14f45460754bccb6398ba5eb51b39f1f161573f51a94438f7b25a003fdf0 WatchSource:0}: Error finding container 215e14f45460754bccb6398ba5eb51b39f1f161573f51a94438f7b25a003fdf0: Status 404 returned error can't find the container with id 215e14f45460754bccb6398ba5eb51b39f1f161573f51a94438f7b25a003fdf0
Jul 29 23:00:19.807844 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:19.807817    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-n4kbt"]
Jul 29 23:00:19.810668 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:00:19.810647    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9778d8_4555_443c_91a5_eabd88c1fccb.slice/crio-f82bd8f620df0f8972a7a4f512f9f09ac9dd63ae9e1e03c09b77224eff079542 WatchSource:0}: Error finding container f82bd8f620df0f8972a7a4f512f9f09ac9dd63ae9e1e03c09b77224eff079542: Status 404 returned error can't find the container with id f82bd8f620df0f8972a7a4f512f9f09ac9dd63ae9e1e03c09b77224eff079542
Jul 29 23:00:20.115026 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:20.114990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-6t2tn" event={"ID":"e42d3a61-ebda-44c1-9c22-deea87c1f921","Type":"ContainerStarted","Data":"215e14f45460754bccb6398ba5eb51b39f1f161573f51a94438f7b25a003fdf0"}
Jul 29 23:00:20.116111 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:20.116087    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-n4kbt" event={"ID":"5e9778d8-4555-443c-91a5-eabd88c1fccb","Type":"ContainerStarted","Data":"f82bd8f620df0f8972a7a4f512f9f09ac9dd63ae9e1e03c09b77224eff079542"}
Jul 29 23:00:24.138356 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:24.138318    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-n4kbt" event={"ID":"5e9778d8-4555-443c-91a5-eabd88c1fccb","Type":"ContainerStarted","Data":"8636f1e9c666fec9f818406b765cd7a2a38927141a8ecb631bddb4b52cd9272e"}
Jul 29 23:00:24.138821 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:24.138436    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-8588677cfb-n4kbt"
Jul 29 23:00:24.139834 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:24.139788    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-6t2tn" event={"ID":"e42d3a61-ebda-44c1-9c22-deea87c1f921","Type":"ContainerStarted","Data":"fd409d90f611af91770125b28de64e1c415ccd15cb818a16eb29f5d2eb33a4e2"}
Jul 29 23:00:24.139934 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:24.139856    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-6t2tn"
Jul 29 23:00:24.161129 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:24.161078    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-8588677cfb-n4kbt" podStartSLOduration=2.503561545 podStartE2EDuration="6.161059209s" totalImagesPullingTime="3.657497664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:00:24.15442611 +0000 UTC m=+846.712770927" watchObservedRunningTime="2026-07-29 23:00:24.161059209 +0000 UTC m=+846.719404027"
Jul 29 23:00:24.172514 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:24.172469    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-6t2tn" podStartSLOduration=2.5065679530000002 podStartE2EDuration="6.172452741s" totalImagesPullingTime="3.665884788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:00:24.17078787 +0000 UTC m=+846.729132685" watchObservedRunningTime="2026-07-29 23:00:24.172452741 +0000 UTC m=+846.730797558"
Jul 29 23:00:35.147111 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:35.147074    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-8588677cfb-n4kbt"
Jul 29 23:00:35.148998 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:35.148980    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-6t2tn"
Jul 29 23:00:35.325034 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:35.324986    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:00:35.425290 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:00:35.425195    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:01:09.481023 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:01:09.480935    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:01:09.539334 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:01:09.539297    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:01:31.759910 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:01:31.759872    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:01:56.198050 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:01:56.198011    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:01:57.857682 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:01:57.857643    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:02:09.552829 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:09.552770    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:02:09.747753 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:09.747718    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:02:22.216311 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:22.216275    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:02:22.555688 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:22.555612    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7"]
Jul 29 23:02:22.559016 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:22.558995    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7"
Jul 29 23:02:22.561947 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:22.561922    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 29 23:02:22.563296 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:22.563190    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 29 23:02:22.563296 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:22.563245    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-fc8tf\""
Jul 29 23:02:22.563296 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:22.563247    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 29 23:02:22.563498 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:22.563275    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-k9vw7\""
Jul 29 23:02:22.567480 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:22.567459    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7"]
Jul 29 23:02:22.590182 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:22.590147    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:02:22.685260 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:22.685231    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n65sc\" (UniqueName: \"kubernetes.io/projected/f6b6a09b-2f8f-41c8-a692-8ad86478a523-kube-api-access-n65sc\") pod \"fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7\" (UID: \"f6b6a09b-2f8f-41c8-a692-8ad86478a523\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7"
Jul 29 23:02:22.685447 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:22.685267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b6a09b-2f8f-41c8-a692-8ad86478a523-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7\" (UID: \"f6b6a09b-2f8f-41c8-a692-8ad86478a523\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7"
Jul 29 23:02:22.786666 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:22.786630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n65sc\" (UniqueName: \"kubernetes.io/projected/f6b6a09b-2f8f-41c8-a692-8ad86478a523-kube-api-access-n65sc\") pod \"fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7\" (UID: \"f6b6a09b-2f8f-41c8-a692-8ad86478a523\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7"
Jul 29 23:02:22.786892 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:22.786677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b6a09b-2f8f-41c8-a692-8ad86478a523-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7\" (UID: \"f6b6a09b-2f8f-41c8-a692-8ad86478a523\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7"
Jul 29 23:02:22.789359 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:22.789323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b6a09b-2f8f-41c8-a692-8ad86478a523-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7\" (UID: \"f6b6a09b-2f8f-41c8-a692-8ad86478a523\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7"
Jul 29 23:02:22.794932 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:22.794908    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n65sc\" (UniqueName: \"kubernetes.io/projected/f6b6a09b-2f8f-41c8-a692-8ad86478a523-kube-api-access-n65sc\") pod \"fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7\" (UID: \"f6b6a09b-2f8f-41c8-a692-8ad86478a523\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7"
Jul 29 23:02:22.871108 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:22.871064    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7"
Jul 29 23:02:23.023214 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:23.023132    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7"]
Jul 29 23:02:23.029048 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:02:23.029018    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b6a09b_2f8f_41c8_a692_8ad86478a523.slice/crio-f5c3b9a39ae353f2a181bc9fe4aeb8306b439d305de11d818bae0155d7b4d44f WatchSource:0}: Error finding container f5c3b9a39ae353f2a181bc9fe4aeb8306b439d305de11d818bae0155d7b4d44f: Status 404 returned error can't find the container with id f5c3b9a39ae353f2a181bc9fe4aeb8306b439d305de11d818bae0155d7b4d44f
Jul 29 23:02:23.031554 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:23.031532    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 23:02:23.674666 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:23.674628    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7" event={"ID":"f6b6a09b-2f8f-41c8-a692-8ad86478a523","Type":"ContainerStarted","Data":"f5c3b9a39ae353f2a181bc9fe4aeb8306b439d305de11d818bae0155d7b4d44f"}
Jul 29 23:02:25.687871 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:25.687835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7" event={"ID":"f6b6a09b-2f8f-41c8-a692-8ad86478a523","Type":"ContainerStarted","Data":"c5ac92893c94c98036d084285415e3e7f3a6e76ae835ab026d1f5f86802570f4"}
Jul 29 23:02:25.688291 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:25.687924    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7"
Jul 29 23:02:25.708325 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:25.708278    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7" podStartSLOduration=1.152387772 podStartE2EDuration="3.708260846s" totalImagesPullingTime="2.555873074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:02:25.706827338 +0000 UTC m=+968.265172156" watchObservedRunningTime="2026-07-29 23:02:25.708260846 +0000 UTC m=+968.266605663"
Jul 29 23:02:27.033443 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:27.033403    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:02:54.233026 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:54.232983    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:02:56.694765 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:02:56.694736    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7"
Jul 29 23:03:01.107581 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:01.107545    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:03:14.320121 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:14.320086    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7"]
Jul 29 23:03:14.323005 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:14.322971    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:03:14.384404 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:14.384377    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:03:16.695037 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:16.694991    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7" podUID="f6b6a09b-2f8f-41c8-a692-8ad86478a523" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:03:26.694697 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:26.694656    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7" podUID="f6b6a09b-2f8f-41c8-a692-8ad86478a523" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:03:29.339444 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:29.339381    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7" podUID="f6b6a09b-2f8f-41c8-a692-8ad86478a523" containerName="main" containerID="cri-o://c5ac92893c94c98036d084285415e3e7f3a6e76ae835ab026d1f5f86802570f4" gracePeriod=45
Jul 29 23:03:29.589404 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:29.589380    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7"
Jul 29 23:03:29.642842 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:29.642777    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b6a09b-2f8f-41c8-a692-8ad86478a523-tls-certs\") pod \"f6b6a09b-2f8f-41c8-a692-8ad86478a523\" (UID: \"f6b6a09b-2f8f-41c8-a692-8ad86478a523\") "
Jul 29 23:03:29.642950 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:29.642872    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n65sc\" (UniqueName: \"kubernetes.io/projected/f6b6a09b-2f8f-41c8-a692-8ad86478a523-kube-api-access-n65sc\") pod \"f6b6a09b-2f8f-41c8-a692-8ad86478a523\" (UID: \"f6b6a09b-2f8f-41c8-a692-8ad86478a523\") "
Jul 29 23:03:29.644838 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:29.644781    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6b6a09b-2f8f-41c8-a692-8ad86478a523-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6b6a09b-2f8f-41c8-a692-8ad86478a523" (UID: "f6b6a09b-2f8f-41c8-a692-8ad86478a523"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:03:29.644838 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:29.644826    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b6a09b-2f8f-41c8-a692-8ad86478a523-kube-api-access-n65sc" (OuterVolumeSpecName: "kube-api-access-n65sc") pod "f6b6a09b-2f8f-41c8-a692-8ad86478a523" (UID: "f6b6a09b-2f8f-41c8-a692-8ad86478a523"). InnerVolumeSpecName "kube-api-access-n65sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:03:29.744114 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:29.744080    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n65sc\" (UniqueName: \"kubernetes.io/projected/f6b6a09b-2f8f-41c8-a692-8ad86478a523-kube-api-access-n65sc\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:03:29.744114 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:29.744112    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b6a09b-2f8f-41c8-a692-8ad86478a523-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:03:29.979856 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:29.979825    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6b6a09b-2f8f-41c8-a692-8ad86478a523" containerID="c5ac92893c94c98036d084285415e3e7f3a6e76ae835ab026d1f5f86802570f4" exitCode=0
Jul 29 23:03:29.979959 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:29.979875    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7"
Jul 29 23:03:29.979959 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:29.979904    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7" event={"ID":"f6b6a09b-2f8f-41c8-a692-8ad86478a523","Type":"ContainerDied","Data":"c5ac92893c94c98036d084285415e3e7f3a6e76ae835ab026d1f5f86802570f4"}
Jul 29 23:03:29.979959 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:29.979951    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7" event={"ID":"f6b6a09b-2f8f-41c8-a692-8ad86478a523","Type":"ContainerDied","Data":"f5c3b9a39ae353f2a181bc9fe4aeb8306b439d305de11d818bae0155d7b4d44f"}
Jul 29 23:03:29.980065 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:29.979974    2581 scope.go:117] "RemoveContainer" containerID="c5ac92893c94c98036d084285415e3e7f3a6e76ae835ab026d1f5f86802570f4"
Jul 29 23:03:29.988588 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:29.988573    2581 scope.go:117] "RemoveContainer" containerID="c5ac92893c94c98036d084285415e3e7f3a6e76ae835ab026d1f5f86802570f4"
Jul 29 23:03:29.988875 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:03:29.988857    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ac92893c94c98036d084285415e3e7f3a6e76ae835ab026d1f5f86802570f4\": container with ID starting with c5ac92893c94c98036d084285415e3e7f3a6e76ae835ab026d1f5f86802570f4 not found: ID does not exist" containerID="c5ac92893c94c98036d084285415e3e7f3a6e76ae835ab026d1f5f86802570f4"
Jul 29 23:03:29.988964 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:29.988885    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ac92893c94c98036d084285415e3e7f3a6e76ae835ab026d1f5f86802570f4"} err="failed to get container status \"c5ac92893c94c98036d084285415e3e7f3a6e76ae835ab026d1f5f86802570f4\": rpc error: code = NotFound desc = could not find container \"c5ac92893c94c98036d084285415e3e7f3a6e76ae835ab026d1f5f86802570f4\": container with ID starting with c5ac92893c94c98036d084285415e3e7f3a6e76ae835ab026d1f5f86802570f4 not found: ID does not exist"
Jul 29 23:03:30.000699 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:30.000677    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7"]
Jul 29 23:03:30.005321 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:30.005304    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5c5b577db-jfbb7"]
Jul 29 23:03:31.953410 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:31.953375    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6b6a09b-2f8f-41c8-a692-8ad86478a523" path="/var/lib/kubelet/pods/f6b6a09b-2f8f-41c8-a692-8ad86478a523/volumes"
Jul 29 23:03:34.360697 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:34.360664    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:03:34.455155 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:34.455121    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:03:40.289418 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.289379    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:03:40.423569 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.423540    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:03:40.638668 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.638638    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7"]
Jul 29 23:03:40.639012 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.638999    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6b6a09b-2f8f-41c8-a692-8ad86478a523" containerName="main"
Jul 29 23:03:40.639062 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.639013    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6b6a09b-2f8f-41c8-a692-8ad86478a523" containerName="main"
Jul 29 23:03:40.639098 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.639094    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6b6a09b-2f8f-41c8-a692-8ad86478a523" containerName="main"
Jul 29 23:03:40.641510 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.641492    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7"
Jul 29 23:03:40.645713 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.645692    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-8pprj\""
Jul 29 23:03:40.645856 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.645707    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-7vgvm\""
Jul 29 23:03:40.645856 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.645711    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Jul 29 23:03:40.645856 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.645691    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Jul 29 23:03:40.645856 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.645726    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Jul 29 23:03:40.649412 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.649393    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7"]
Jul 29 23:03:40.666584 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.666545    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:03:40.722690 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.722667    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-298ch\" (UniqueName: \"kubernetes.io/projected/da4bd64b-1c08-4cf2-820a-d86afaa39890-kube-api-access-298ch\") pod \"fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7\" (UID: \"da4bd64b-1c08-4cf2-820a-d86afaa39890\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7"
Jul 29 23:03:40.722794 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.722744    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4bd64b-1c08-4cf2-820a-d86afaa39890-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7\" (UID: \"da4bd64b-1c08-4cf2-820a-d86afaa39890\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7"
Jul 29 23:03:40.823308 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.823272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-298ch\" (UniqueName: \"kubernetes.io/projected/da4bd64b-1c08-4cf2-820a-d86afaa39890-kube-api-access-298ch\") pod \"fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7\" (UID: \"da4bd64b-1c08-4cf2-820a-d86afaa39890\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7"
Jul 29 23:03:40.823424 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.823348    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4bd64b-1c08-4cf2-820a-d86afaa39890-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7\" (UID: \"da4bd64b-1c08-4cf2-820a-d86afaa39890\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7"
Jul 29 23:03:40.825608 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.825591    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4bd64b-1c08-4cf2-820a-d86afaa39890-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7\" (UID: \"da4bd64b-1c08-4cf2-820a-d86afaa39890\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7"
Jul 29 23:03:40.830947 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.830931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-298ch\" (UniqueName: \"kubernetes.io/projected/da4bd64b-1c08-4cf2-820a-d86afaa39890-kube-api-access-298ch\") pod \"fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7\" (UID: \"da4bd64b-1c08-4cf2-820a-d86afaa39890\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7"
Jul 29 23:03:40.952400 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:40.952329    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7"
Jul 29 23:03:41.081501 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:41.081429    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7"]
Jul 29 23:03:41.084460 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:03:41.084432    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4bd64b_1c08_4cf2_820a_d86afaa39890.slice/crio-1ad2c8ae9bb8c99e805564916f9453b9fc19a1ec23222738a04296bce4ef5b0d WatchSource:0}: Error finding container 1ad2c8ae9bb8c99e805564916f9453b9fc19a1ec23222738a04296bce4ef5b0d: Status 404 returned error can't find the container with id 1ad2c8ae9bb8c99e805564916f9453b9fc19a1ec23222738a04296bce4ef5b0d
Jul 29 23:03:42.034895 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:42.034867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7" event={"ID":"da4bd64b-1c08-4cf2-820a-d86afaa39890","Type":"ContainerStarted","Data":"51d9b3e3b1475a619e418b22d167d05de0edc0ce5db26b77cebdd8f477e641ef"}
Jul 29 23:03:42.035261 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:42.034911    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7" event={"ID":"da4bd64b-1c08-4cf2-820a-d86afaa39890","Type":"ContainerStarted","Data":"1ad2c8ae9bb8c99e805564916f9453b9fc19a1ec23222738a04296bce4ef5b0d"}
Jul 29 23:03:42.035261 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:42.034949    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7"
Jul 29 23:03:42.052579 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:42.052533    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7" podStartSLOduration=2.052519568 podStartE2EDuration="2.052519568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:03:42.05057135 +0000 UTC m=+1044.608916160" watchObservedRunningTime="2026-07-29 23:03:42.052519568 +0000 UTC m=+1044.610864421"
Jul 29 23:03:45.843474 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:45.843432    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:03:45.905366 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:45.905333    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:03:45.995419 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:45.995379    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:03:56.733169 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:03:56.733124    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:04:13.040232 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:13.040140    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7"
Jul 29 23:04:19.331993 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:19.331949    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7"]
Jul 29 23:04:19.372910 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:19.372875    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:04:19.471400 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:19.471363    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:04:23.041027 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:23.040989    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7" podUID="da4bd64b-1c08-4cf2-820a-d86afaa39890" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:04:33.040929 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:33.040891    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7" podUID="da4bd64b-1c08-4cf2-820a-d86afaa39890" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:04:34.352398 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:34.352346    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7" podUID="da4bd64b-1c08-4cf2-820a-d86afaa39890" containerName="main" containerID="cri-o://51d9b3e3b1475a619e418b22d167d05de0edc0ce5db26b77cebdd8f477e641ef" gracePeriod=45
Jul 29 23:04:34.602820 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:34.602734    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7"
Jul 29 23:04:34.691123 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:34.691088    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4bd64b-1c08-4cf2-820a-d86afaa39890-tls-certs\") pod \"da4bd64b-1c08-4cf2-820a-d86afaa39890\" (UID: \"da4bd64b-1c08-4cf2-820a-d86afaa39890\") "
Jul 29 23:04:34.691313 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:34.691134    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-298ch\" (UniqueName: \"kubernetes.io/projected/da4bd64b-1c08-4cf2-820a-d86afaa39890-kube-api-access-298ch\") pod \"da4bd64b-1c08-4cf2-820a-d86afaa39890\" (UID: \"da4bd64b-1c08-4cf2-820a-d86afaa39890\") "
Jul 29 23:04:34.693383 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:34.693339    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4bd64b-1c08-4cf2-820a-d86afaa39890-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da4bd64b-1c08-4cf2-820a-d86afaa39890" (UID: "da4bd64b-1c08-4cf2-820a-d86afaa39890"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:04:34.693501 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:34.693387    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4bd64b-1c08-4cf2-820a-d86afaa39890-kube-api-access-298ch" (OuterVolumeSpecName: "kube-api-access-298ch") pod "da4bd64b-1c08-4cf2-820a-d86afaa39890" (UID: "da4bd64b-1c08-4cf2-820a-d86afaa39890"). InnerVolumeSpecName "kube-api-access-298ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:04:34.792571 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:34.792536    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4bd64b-1c08-4cf2-820a-d86afaa39890-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:04:34.792571 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:34.792564    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-298ch\" (UniqueName: \"kubernetes.io/projected/da4bd64b-1c08-4cf2-820a-d86afaa39890-kube-api-access-298ch\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:04:35.281952 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:35.281920    2581 generic.go:356] "Generic (PLEG): container finished" podID="da4bd64b-1c08-4cf2-820a-d86afaa39890" containerID="51d9b3e3b1475a619e418b22d167d05de0edc0ce5db26b77cebdd8f477e641ef" exitCode=0
Jul 29 23:04:35.282123 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:35.281970    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7"
Jul 29 23:04:35.282123 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:35.281971    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7" event={"ID":"da4bd64b-1c08-4cf2-820a-d86afaa39890","Type":"ContainerDied","Data":"51d9b3e3b1475a619e418b22d167d05de0edc0ce5db26b77cebdd8f477e641ef"}
Jul 29 23:04:35.282123 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:35.282100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7" event={"ID":"da4bd64b-1c08-4cf2-820a-d86afaa39890","Type":"ContainerDied","Data":"1ad2c8ae9bb8c99e805564916f9453b9fc19a1ec23222738a04296bce4ef5b0d"}
Jul 29 23:04:35.282242 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:35.282133    2581 scope.go:117] "RemoveContainer" containerID="51d9b3e3b1475a619e418b22d167d05de0edc0ce5db26b77cebdd8f477e641ef"
Jul 29 23:04:35.291464 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:35.291445    2581 scope.go:117] "RemoveContainer" containerID="51d9b3e3b1475a619e418b22d167d05de0edc0ce5db26b77cebdd8f477e641ef"
Jul 29 23:04:35.291734 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:04:35.291711    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d9b3e3b1475a619e418b22d167d05de0edc0ce5db26b77cebdd8f477e641ef\": container with ID starting with 51d9b3e3b1475a619e418b22d167d05de0edc0ce5db26b77cebdd8f477e641ef not found: ID does not exist" containerID="51d9b3e3b1475a619e418b22d167d05de0edc0ce5db26b77cebdd8f477e641ef"
Jul 29 23:04:35.291779 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:35.291744    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d9b3e3b1475a619e418b22d167d05de0edc0ce5db26b77cebdd8f477e641ef"} err="failed to get container status \"51d9b3e3b1475a619e418b22d167d05de0edc0ce5db26b77cebdd8f477e641ef\": rpc error: code = NotFound desc = could not find container \"51d9b3e3b1475a619e418b22d167d05de0edc0ce5db26b77cebdd8f477e641ef\": container with ID starting with 51d9b3e3b1475a619e418b22d167d05de0edc0ce5db26b77cebdd8f477e641ef not found: ID does not exist"
Jul 29 23:04:35.304545 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:35.304520    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7"]
Jul 29 23:04:35.307767 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:35.307747    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-b455bf55c-jf7v7"]
Jul 29 23:04:35.952542 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:35.952509    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da4bd64b-1c08-4cf2-820a-d86afaa39890" path="/var/lib/kubelet/pods/da4bd64b-1c08-4cf2-820a-d86afaa39890/volumes"
Jul 29 23:04:39.246130 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.246097    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:04:39.376091 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.376061    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:04:39.555848 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.555737    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"]
Jul 29 23:04:39.556335 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.556313    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da4bd64b-1c08-4cf2-820a-d86afaa39890" containerName="main"
Jul 29 23:04:39.556404 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.556339    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="da4bd64b-1c08-4cf2-820a-d86afaa39890" containerName="main"
Jul 29 23:04:39.556483 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.556470    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="da4bd64b-1c08-4cf2-820a-d86afaa39890" containerName="main"
Jul 29 23:04:39.560568 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.560548    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.563628 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.563601    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Jul 29 23:04:39.563762 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.563738    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Jul 29 23:04:39.563865 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.563742    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Jul 29 23:04:39.565079 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.565060    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-hn5tg\""
Jul 29 23:04:39.576554 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.576531    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"]
Jul 29 23:04:39.586517 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.586494    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:04:39.630251 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.630220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.630381 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.630295    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-data\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.630381 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.630325    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-podinfo\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.630381 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.630344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-token\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.630381 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.630367    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj6jc\" (UniqueName: \"kubernetes.io/projected/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-kube-api-access-xj6jc\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.630604 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.630413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-envoy\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.630604 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.630495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-credential-socket\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.630604 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.630518    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-workload-certs\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.630604 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.630538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-workload-socket\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.686598 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.686569    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:04:39.731903 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.731881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.732012 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.731913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-data\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.732012 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.731934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-podinfo\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.732012 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.731959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-token\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.732012 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.731982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj6jc\" (UniqueName: \"kubernetes.io/projected/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-kube-api-access-xj6jc\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.732012 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.732007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-envoy\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.732230 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.732050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-credential-socket\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.732230 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.732078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-workload-certs\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.732230 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.732111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-workload-socket\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.732380 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.732308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-data\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.732483 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.732466    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-credential-socket\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.732533 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.732509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-workload-socket\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.732578 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.732556    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-workload-certs\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.732830 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.732793    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.734159 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.734142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-envoy\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.734741 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.734726    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-podinfo\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.740943 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.740925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-token\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.741143 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.741126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj6jc\" (UniqueName: \"kubernetes.io/projected/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-kube-api-access-xj6jc\") pod \"router-gateway-1-openshift-default-55764c768d-vztzn\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:39.871214 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:39.871195    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:40.030100 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:40.030076    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"]
Jul 29 23:04:40.032703 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:04:40.032676    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c33c2e_cad9_4354_b651_e8c14bf4a0ab.slice/crio-1afb477294a925dc251702251e5231ccf09a53ca26c16301e1009feb47e7e1ed WatchSource:0}: Error finding container 1afb477294a925dc251702251e5231ccf09a53ca26c16301e1009feb47e7e1ed: Status 404 returned error can't find the container with id 1afb477294a925dc251702251e5231ccf09a53ca26c16301e1009feb47e7e1ed
Jul 29 23:04:40.307433 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:40.307358    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn" event={"ID":"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab","Type":"ContainerStarted","Data":"1afb477294a925dc251702251e5231ccf09a53ca26c16301e1009feb47e7e1ed"}
Jul 29 23:04:42.492686 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:42.492649    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:04:44.823047 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:44.822998    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 23:04:44.823308 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:44.823088    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 23:04:44.823308 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:44.823115    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 23:04:45.260556 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:45.260511    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:04:45.332961 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:45.332932    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn" event={"ID":"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab","Type":"ContainerStarted","Data":"72b658d7952a51ad00a13d6f3ef035cc27a7e6884212b2ab6aeefb220d76bda4"}
Jul 29 23:04:45.356360 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:45.356307    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn" podStartSLOduration=1.568502992 podStartE2EDuration="6.356291813s" totalImagesPullingTime="4.787788821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:04:45.354104715 +0000 UTC m=+1107.912449548" watchObservedRunningTime="2026-07-29 23:04:45.356291813 +0000 UTC m=+1107.914636630"
Jul 29 23:04:45.872396 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:45.872366    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:45.877250 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:45.877227    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:46.337899 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:46.337867    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:46.339046 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:46.339026    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:04:50.776365 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:50.776330    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:04:50.922735 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:04:50.922702    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:00.321815 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:00.321758    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:01.534595 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:01.534552    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:02.112001 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.111959    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:02.214186 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.214147    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:02.324126 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.324083    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:02.480616 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.480519    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"]
Jul 29 23:05:02.484507 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.484482    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.487721 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.487695    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Jul 29 23:05:02.487887 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.487746    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Jul 29 23:05:02.487980 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.487880    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-zdjxg\""
Jul 29 23:05:02.487980 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.487891    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Jul 29 23:05:02.497195 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.497175    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"]
Jul 29 23:05:02.522179 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.522151    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:02.621838 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.621792    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:02.636046 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.636007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-envoy\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.636203 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.636075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-workload-socket\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.636203 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.636180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-credential-socket\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.636301 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.636222    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-data\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.636301 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.636245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-token\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.636395 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.636336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-workload-certs\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.636395 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.636364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.636483 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.636419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-podinfo\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.636541 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.636524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc9sf\" (UniqueName: \"kubernetes.io/projected/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-kube-api-access-lc9sf\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.737501 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.737412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-workload-certs\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.737501 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.737447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.737501 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.737470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-podinfo\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.737770 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.737516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc9sf\" (UniqueName: \"kubernetes.io/projected/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-kube-api-access-lc9sf\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.737770 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.737554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-envoy\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.737770 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.737603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-workload-socket\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.737770 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.737628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-credential-socket\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.737770 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.737652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-data\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.737770 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.737674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-token\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.738100 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.737848    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-workload-certs\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.738100 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.737992    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-workload-socket\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.738100 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.738084    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-credential-socket\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.738100 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.738093    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-data\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.738254 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.738237    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.740054 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.740034    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-envoy\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.740226 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.740211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-podinfo\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.745943 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.745913    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-token\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.746078 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.746064    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc9sf\" (UniqueName: \"kubernetes.io/projected/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-kube-api-access-lc9sf\") pod \"router-gateway-1-openshift-default-54df766bd8-vbkp4\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.797649 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.797615    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:02.969833 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.963042    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"]
Jul 29 23:05:02.980274 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:05:02.977148    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4685fa4b_e0c8_4b43_ab69_b4766b7c3fd9.slice/crio-c0aaa15e688d5fcf44f4b24e2106a567cfe6cdb44cfcbcde1235d0377a2a1b5e WatchSource:0}: Error finding container c0aaa15e688d5fcf44f4b24e2106a567cfe6cdb44cfcbcde1235d0377a2a1b5e: Status 404 returned error can't find the container with id c0aaa15e688d5fcf44f4b24e2106a567cfe6cdb44cfcbcde1235d0377a2a1b5e
Jul 29 23:05:02.982692 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.982660    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 23:05:02.982785 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.982758    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 23:05:02.982952 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:02.982828    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 23:05:03.419228 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:03.419192    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4" event={"ID":"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9","Type":"ContainerStarted","Data":"fc97706c5f64c6c74ba5719626b37043067ca839e80ecabd1145c83aa29c64a6"}
Jul 29 23:05:03.419403 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:03.419272    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4" event={"ID":"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9","Type":"ContainerStarted","Data":"c0aaa15e688d5fcf44f4b24e2106a567cfe6cdb44cfcbcde1235d0377a2a1b5e"}
Jul 29 23:05:03.441923 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:03.441873    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4" podStartSLOduration=1.441858586 podStartE2EDuration="1.441858586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:05:03.440070504 +0000 UTC m=+1125.998415317" watchObservedRunningTime="2026-07-29 23:05:03.441858586 +0000 UTC m=+1126.000203401"
Jul 29 23:05:03.797970 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:03.797886    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:03.936365 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:03.936324    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:04.803698 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:04.803665    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:05.428765 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:05.428729    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:05.429751 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:05.429731    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:05.620849 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:05.620818    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:07.091889 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:07.091854    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"]
Jul 29 23:05:07.092314 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:07.092114    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn" podUID="d5c33c2e-cad9-4354-b651-e8c14bf4a0ab" containerName="istio-proxy" containerID="cri-o://72b658d7952a51ad00a13d6f3ef035cc27a7e6884212b2ab6aeefb220d76bda4" gracePeriod=30
Jul 29 23:05:07.476054 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:07.475970    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:09.682698 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.682665    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:09.738068 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.738048    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:05:09.869455 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.869389    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:09.906386 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.906361    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj6jc\" (UniqueName: \"kubernetes.io/projected/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-kube-api-access-xj6jc\") pod \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") "
Jul 29 23:05:09.906497 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.906404    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-envoy\") pod \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") "
Jul 29 23:05:09.906497 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.906423    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-workload-socket\") pod \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") "
Jul 29 23:05:09.906497 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.906442    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-workload-certs\") pod \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") "
Jul 29 23:05:09.906665 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.906498    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-podinfo\") pod \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") "
Jul 29 23:05:09.906665 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.906518    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-credential-socket\") pod \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") "
Jul 29 23:05:09.906665 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.906556    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istiod-ca-cert\") pod \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") "
Jul 29 23:05:09.906665 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.906583    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-token\") pod \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") "
Jul 29 23:05:09.906665 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.906608    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-data\") pod \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\" (UID: \"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab\") "
Jul 29 23:05:09.906975 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.906688    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d5c33c2e-cad9-4354-b651-e8c14bf4a0ab" (UID: "d5c33c2e-cad9-4354-b651-e8c14bf4a0ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:09.906975 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.906893    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d5c33c2e-cad9-4354-b651-e8c14bf4a0ab" (UID: "d5c33c2e-cad9-4354-b651-e8c14bf4a0ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:09.906975 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.906935    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-workload-socket\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:05:09.907131 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.907076    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-data" (OuterVolumeSpecName: "istio-data") pod "d5c33c2e-cad9-4354-b651-e8c14bf4a0ab" (UID: "d5c33c2e-cad9-4354-b651-e8c14bf4a0ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:09.907193 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.907125    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d5c33c2e-cad9-4354-b651-e8c14bf4a0ab" (UID: "d5c33c2e-cad9-4354-b651-e8c14bf4a0ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 23:05:09.907247 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.907218    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d5c33c2e-cad9-4354-b651-e8c14bf4a0ab" (UID: "d5c33c2e-cad9-4354-b651-e8c14bf4a0ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:09.908675 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.908648    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d5c33c2e-cad9-4354-b651-e8c14bf4a0ab" (UID: "d5c33c2e-cad9-4354-b651-e8c14bf4a0ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:09.908776 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.908686    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d5c33c2e-cad9-4354-b651-e8c14bf4a0ab" (UID: "d5c33c2e-cad9-4354-b651-e8c14bf4a0ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 23:05:09.908776 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.908769    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-kube-api-access-xj6jc" (OuterVolumeSpecName: "kube-api-access-xj6jc") pod "d5c33c2e-cad9-4354-b651-e8c14bf4a0ab" (UID: "d5c33c2e-cad9-4354-b651-e8c14bf4a0ab"). InnerVolumeSpecName "kube-api-access-xj6jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:05:09.908900 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:09.908822    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-token" (OuterVolumeSpecName: "istio-token") pod "d5c33c2e-cad9-4354-b651-e8c14bf4a0ab" (UID: "d5c33c2e-cad9-4354-b651-e8c14bf4a0ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:05:10.007640 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:10.007618    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-podinfo\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:05:10.007640 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:10.007638    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-credential-socket\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:05:10.007862 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:10.007649    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istiod-ca-cert\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:05:10.007862 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:10.007658    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-token\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:05:10.007862 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:10.007666    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-data\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:05:10.007862 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:10.007674    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj6jc\" (UniqueName: \"kubernetes.io/projected/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-kube-api-access-xj6jc\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:05:10.007862 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:10.007681    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-istio-envoy\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:05:10.007862 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:10.007689    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab-workload-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:05:10.454839 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:10.454784    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5c33c2e-cad9-4354-b651-e8c14bf4a0ab" containerID="72b658d7952a51ad00a13d6f3ef035cc27a7e6884212b2ab6aeefb220d76bda4" exitCode=0
Jul 29 23:05:10.454839 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:10.454830    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn" event={"ID":"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab","Type":"ContainerDied","Data":"72b658d7952a51ad00a13d6f3ef035cc27a7e6884212b2ab6aeefb220d76bda4"}
Jul 29 23:05:10.455053 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:10.454849    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"
Jul 29 23:05:10.455053 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:10.454885    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn" event={"ID":"d5c33c2e-cad9-4354-b651-e8c14bf4a0ab","Type":"ContainerDied","Data":"1afb477294a925dc251702251e5231ccf09a53ca26c16301e1009feb47e7e1ed"}
Jul 29 23:05:10.455053 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:10.454912    2581 scope.go:117] "RemoveContainer" containerID="72b658d7952a51ad00a13d6f3ef035cc27a7e6884212b2ab6aeefb220d76bda4"
Jul 29 23:05:10.463540 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:10.463522    2581 scope.go:117] "RemoveContainer" containerID="72b658d7952a51ad00a13d6f3ef035cc27a7e6884212b2ab6aeefb220d76bda4"
Jul 29 23:05:10.463788 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:05:10.463769    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b658d7952a51ad00a13d6f3ef035cc27a7e6884212b2ab6aeefb220d76bda4\": container with ID starting with 72b658d7952a51ad00a13d6f3ef035cc27a7e6884212b2ab6aeefb220d76bda4 not found: ID does not exist" containerID="72b658d7952a51ad00a13d6f3ef035cc27a7e6884212b2ab6aeefb220d76bda4"
Jul 29 23:05:10.463862 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:10.463818    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b658d7952a51ad00a13d6f3ef035cc27a7e6884212b2ab6aeefb220d76bda4"} err="failed to get container status \"72b658d7952a51ad00a13d6f3ef035cc27a7e6884212b2ab6aeefb220d76bda4\": rpc error: code = NotFound desc = could not find container \"72b658d7952a51ad00a13d6f3ef035cc27a7e6884212b2ab6aeefb220d76bda4\": container with ID starting with 72b658d7952a51ad00a13d6f3ef035cc27a7e6884212b2ab6aeefb220d76bda4 not found: ID does not exist"
Jul 29 23:05:10.476408 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:10.476388    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"]
Jul 29 23:05:10.480923 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:10.480901    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-55764c768d-vztzn"]
Jul 29 23:05:11.953360 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:11.953327    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5c33c2e-cad9-4354-b651-e8c14bf4a0ab" path="/var/lib/kubelet/pods/d5c33c2e-cad9-4354-b651-e8c14bf4a0ab/volumes"
Jul 29 23:05:12.251471 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:12.251379    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:13.104172 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:13.104139    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:15.219451 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:15.219415    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:17.030704 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:17.030665    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:17.125879 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:17.125841    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:17.589340 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:17.589296    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:18.595070 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:18.595038    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:22.006056 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:22.006024    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"]
Jul 29 23:05:22.006427 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:22.006237    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4" podUID="4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9" containerName="istio-proxy" containerID="cri-o://fc97706c5f64c6c74ba5719626b37043067ca839e80ecabd1145c83aa29c64a6" gracePeriod=30
Jul 29 23:05:22.103669 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:22.103637    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:25.054814 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.054772    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:25.133786 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.133698    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-data\") pod \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") "
Jul 29 23:05:25.133786 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.133759    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istiod-ca-cert\") pod \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") "
Jul 29 23:05:25.134002 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.133840    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-credential-socket\") pod \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") "
Jul 29 23:05:25.134002 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.133878    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc9sf\" (UniqueName: \"kubernetes.io/projected/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-kube-api-access-lc9sf\") pod \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") "
Jul 29 23:05:25.134002 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.133919    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-envoy\") pod \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") "
Jul 29 23:05:25.134002 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.133948    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-podinfo\") pod \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") "
Jul 29 23:05:25.134002 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.133976    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-workload-socket\") pod \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") "
Jul 29 23:05:25.134002 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.133978    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-data" (OuterVolumeSpecName: "istio-data") pod "4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9" (UID: "4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:25.134314 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.134008    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-token\") pod \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") "
Jul 29 23:05:25.134314 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.134034    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-workload-certs\") pod \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\" (UID: \"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9\") "
Jul 29 23:05:25.134314 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.134118    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9" (UID: "4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:25.134314 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.134224    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9" (UID: "4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:25.134314 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.134271    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9" (UID: "4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 23:05:25.134573 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.134319    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-workload-socket\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:05:25.134573 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.134325    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9" (UID: "4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:25.134573 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.134339    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-data\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:05:25.134573 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.134354    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-credential-socket\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:05:25.136242 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.136215    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9" (UID: "4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 23:05:25.136536 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.136506    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-token" (OuterVolumeSpecName: "istio-token") pod "4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9" (UID: "4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:05:25.136653 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.136609    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-kube-api-access-lc9sf" (OuterVolumeSpecName: "kube-api-access-lc9sf") pod "4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9" (UID: "4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9"). InnerVolumeSpecName "kube-api-access-lc9sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:05:25.136705 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.136647    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9" (UID: "4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:25.234959 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.234925    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istiod-ca-cert\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:05:25.234959 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.234952    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc9sf\" (UniqueName: \"kubernetes.io/projected/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-kube-api-access-lc9sf\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:05:25.234959 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.234963    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-envoy\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:05:25.235182 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.234972    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-podinfo\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:05:25.235182 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.234979    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-istio-token\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:05:25.235182 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.234988    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9-workload-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:05:25.531956 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.531868    2581 generic.go:356] "Generic (PLEG): container finished" podID="4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9" containerID="fc97706c5f64c6c74ba5719626b37043067ca839e80ecabd1145c83aa29c64a6" exitCode=0
Jul 29 23:05:25.531956 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.531911    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"
Jul 29 23:05:25.532140 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.531957    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4" event={"ID":"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9","Type":"ContainerDied","Data":"fc97706c5f64c6c74ba5719626b37043067ca839e80ecabd1145c83aa29c64a6"}
Jul 29 23:05:25.532140 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.532007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4" event={"ID":"4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9","Type":"ContainerDied","Data":"c0aaa15e688d5fcf44f4b24e2106a567cfe6cdb44cfcbcde1235d0377a2a1b5e"}
Jul 29 23:05:25.532140 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.532030    2581 scope.go:117] "RemoveContainer" containerID="fc97706c5f64c6c74ba5719626b37043067ca839e80ecabd1145c83aa29c64a6"
Jul 29 23:05:25.541516 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.541499    2581 scope.go:117] "RemoveContainer" containerID="fc97706c5f64c6c74ba5719626b37043067ca839e80ecabd1145c83aa29c64a6"
Jul 29 23:05:25.541761 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:05:25.541742    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc97706c5f64c6c74ba5719626b37043067ca839e80ecabd1145c83aa29c64a6\": container with ID starting with fc97706c5f64c6c74ba5719626b37043067ca839e80ecabd1145c83aa29c64a6 not found: ID does not exist" containerID="fc97706c5f64c6c74ba5719626b37043067ca839e80ecabd1145c83aa29c64a6"
Jul 29 23:05:25.541832 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.541769    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc97706c5f64c6c74ba5719626b37043067ca839e80ecabd1145c83aa29c64a6"} err="failed to get container status \"fc97706c5f64c6c74ba5719626b37043067ca839e80ecabd1145c83aa29c64a6\": rpc error: code = NotFound desc = could not find container \"fc97706c5f64c6c74ba5719626b37043067ca839e80ecabd1145c83aa29c64a6\": container with ID starting with fc97706c5f64c6c74ba5719626b37043067ca839e80ecabd1145c83aa29c64a6 not found: ID does not exist"
Jul 29 23:05:25.555687 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.555664    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"]
Jul 29 23:05:25.560495 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.560476    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-vbkp4"]
Jul 29 23:05:25.704210 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.704174    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:25.952760 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:25.952729    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9" path="/var/lib/kubelet/pods/4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9/volumes"
Jul 29 23:05:27.348468 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:27.348433    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:27.814631 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:27.814600    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:28.087389 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.087350    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:28.327297 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.327265    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp"]
Jul 29 23:05:28.327631 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.327618    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9" containerName="istio-proxy"
Jul 29 23:05:28.327675 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.327633    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9" containerName="istio-proxy"
Jul 29 23:05:28.327675 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.327649    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c33c2e-cad9-4354-b651-e8c14bf4a0ab" containerName="istio-proxy"
Jul 29 23:05:28.327675 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.327658    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c33c2e-cad9-4354-b651-e8c14bf4a0ab" containerName="istio-proxy"
Jul 29 23:05:28.327768 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.327724    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4685fa4b-e0c8-4b43-ab69-b4766b7c3fd9" containerName="istio-proxy"
Jul 29 23:05:28.327768 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.327736    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c33c2e-cad9-4354-b651-e8c14bf4a0ab" containerName="istio-proxy"
Jul 29 23:05:28.330460 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.330442    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp"
Jul 29 23:05:28.334759 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.334734    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-k225s\""
Jul 29 23:05:28.334759 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.334755    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Jul 29 23:05:28.334950 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.334772    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Jul 29 23:05:28.334950 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.334757    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-zpkbs\""
Jul 29 23:05:28.334950 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.334891    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 29 23:05:28.338614 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.338560    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp"]
Jul 29 23:05:28.354637 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.354613    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:28.358674 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.358655    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc3135d-b991-4cf7-8472-6ae44e6c90e0-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp\" (UID: \"0dc3135d-b991-4cf7-8472-6ae44e6c90e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp"
Jul 29 23:05:28.358775 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.358710    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgqrn\" (UniqueName: \"kubernetes.io/projected/0dc3135d-b991-4cf7-8472-6ae44e6c90e0-kube-api-access-jgqrn\") pod \"auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp\" (UID: \"0dc3135d-b991-4cf7-8472-6ae44e6c90e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp"
Jul 29 23:05:28.459261 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.459232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc3135d-b991-4cf7-8472-6ae44e6c90e0-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp\" (UID: \"0dc3135d-b991-4cf7-8472-6ae44e6c90e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp"
Jul 29 23:05:28.459419 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.459284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgqrn\" (UniqueName: \"kubernetes.io/projected/0dc3135d-b991-4cf7-8472-6ae44e6c90e0-kube-api-access-jgqrn\") pod \"auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp\" (UID: \"0dc3135d-b991-4cf7-8472-6ae44e6c90e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp"
Jul 29 23:05:28.461656 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.461629    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc3135d-b991-4cf7-8472-6ae44e6c90e0-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp\" (UID: \"0dc3135d-b991-4cf7-8472-6ae44e6c90e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp"
Jul 29 23:05:28.467869 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.467849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgqrn\" (UniqueName: \"kubernetes.io/projected/0dc3135d-b991-4cf7-8472-6ae44e6c90e0-kube-api-access-jgqrn\") pod \"auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp\" (UID: \"0dc3135d-b991-4cf7-8472-6ae44e6c90e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp"
Jul 29 23:05:28.642050 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.641972    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp"
Jul 29 23:05:28.769355 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.769216    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp"]
Jul 29 23:05:28.771986 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:05:28.771961    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc3135d_b991_4cf7_8472_6ae44e6c90e0.slice/crio-4b4236a02e11e070208daa74e41c633a480bd5a1ace451de2b5956f3a8166370 WatchSource:0}: Error finding container 4b4236a02e11e070208daa74e41c633a480bd5a1ace451de2b5956f3a8166370: Status 404 returned error can't find the container with id 4b4236a02e11e070208daa74e41c633a480bd5a1ace451de2b5956f3a8166370
Jul 29 23:05:28.933595 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:28.933480    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:29.264173 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:29.264086    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:29.552918 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:29.552838    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp" event={"ID":"0dc3135d-b991-4cf7-8472-6ae44e6c90e0","Type":"ContainerStarted","Data":"fb1690bb2d83059bb04e3be15a344d70467ce8a2281a8b9a0e8a50a3f6424068"}
Jul 29 23:05:29.552918 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:29.552892    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp" event={"ID":"0dc3135d-b991-4cf7-8472-6ae44e6c90e0","Type":"ContainerStarted","Data":"4b4236a02e11e070208daa74e41c633a480bd5a1ace451de2b5956f3a8166370"}
Jul 29 23:05:29.553306 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:29.552998    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp"
Jul 29 23:05:29.571292 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:29.571235    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp" podStartSLOduration=1.571215632 podStartE2EDuration="1.571215632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:05:29.569750424 +0000 UTC m=+1152.128095240" watchObservedRunningTime="2026-07-29 23:05:29.571215632 +0000 UTC m=+1152.129560449"
Jul 29 23:05:30.630459 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:30.630424    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:33.812934 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:33.812894    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:37.215924 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:37.215889    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:42.069752 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:42.069667    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:42.171215 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:42.171174    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:51.239447 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:51.239407    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:51.333082 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:51.333051    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:05:57.308505 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:05:57.308463    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:06:00.560104 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:00.560071    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp"
Jul 29 23:06:02.791014 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:02.790979    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:06:02.924309 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:02.924270    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:06:03.046848 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.046752    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:06:03.117815 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.117765    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv"]
Jul 29 23:06:03.121168 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.121151    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv"
Jul 29 23:06:03.126183 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.126150    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 29 23:06:03.126319 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.126198    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 29 23:06:03.126319 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.126150    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-ncx5k\""
Jul 29 23:06:03.126319 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.126240    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-lhrnv\""
Jul 29 23:06:03.126446 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.126392    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 29 23:06:03.139624 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.139600    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv"]
Jul 29 23:06:03.169952 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.169921    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:06:03.236893 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.236858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp5hd\" (UniqueName: \"kubernetes.io/projected/4982a45e-0edb-4713-ab54-ea4526f41fd8-kube-api-access-bp5hd\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv\" (UID: \"4982a45e-0edb-4713-ab54-ea4526f41fd8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv"
Jul 29 23:06:03.237059 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.236960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4982a45e-0edb-4713-ab54-ea4526f41fd8-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv\" (UID: \"4982a45e-0edb-4713-ab54-ea4526f41fd8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv"
Jul 29 23:06:03.337587 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.337557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp5hd\" (UniqueName: \"kubernetes.io/projected/4982a45e-0edb-4713-ab54-ea4526f41fd8-kube-api-access-bp5hd\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv\" (UID: \"4982a45e-0edb-4713-ab54-ea4526f41fd8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv"
Jul 29 23:06:03.337749 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.337644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4982a45e-0edb-4713-ab54-ea4526f41fd8-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv\" (UID: \"4982a45e-0edb-4713-ab54-ea4526f41fd8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv"
Jul 29 23:06:03.340164 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.340142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4982a45e-0edb-4713-ab54-ea4526f41fd8-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv\" (UID: \"4982a45e-0edb-4713-ab54-ea4526f41fd8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv"
Jul 29 23:06:03.347023 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.346989    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp5hd\" (UniqueName: \"kubernetes.io/projected/4982a45e-0edb-4713-ab54-ea4526f41fd8-kube-api-access-bp5hd\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv\" (UID: \"4982a45e-0edb-4713-ab54-ea4526f41fd8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv"
Jul 29 23:06:03.432823 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.432774    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv"
Jul 29 23:06:03.522874 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.522829    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:06:03.576268 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.576237    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv"]
Jul 29 23:06:03.578745 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:06:03.578706    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4982a45e_0edb_4713_ab54_ea4526f41fd8.slice/crio-3ceb20fd9a2b5b596780db81232e20998cdeaab86edd7fb6ca6a95f7d8032f53 WatchSource:0}: Error finding container 3ceb20fd9a2b5b596780db81232e20998cdeaab86edd7fb6ca6a95f7d8032f53: Status 404 returned error can't find the container with id 3ceb20fd9a2b5b596780db81232e20998cdeaab86edd7fb6ca6a95f7d8032f53
Jul 29 23:06:03.624040 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.623975    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:06:03.714667 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.714630    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv" event={"ID":"4982a45e-0edb-4713-ab54-ea4526f41fd8","Type":"ContainerStarted","Data":"65d8a8c484478989527c15778898d8ed3fa34d995057264c96261dadc261b859"}
Jul 29 23:06:03.714883 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.714695    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv" event={"ID":"4982a45e-0edb-4713-ab54-ea4526f41fd8","Type":"ContainerStarted","Data":"3ceb20fd9a2b5b596780db81232e20998cdeaab86edd7fb6ca6a95f7d8032f53"}
Jul 29 23:06:03.714883 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.714846    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv"
Jul 29 23:06:03.745110 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:03.745062    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv" podStartSLOduration=0.745047449 podStartE2EDuration="745.047449ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:06:03.740473058 +0000 UTC m=+1186.298817874" watchObservedRunningTime="2026-07-29 23:06:03.745047449 +0000 UTC m=+1186.303392266"
Jul 29 23:06:13.259415 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:13.259381    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:06:34.727178 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:06:34.727147    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv"
Jul 29 23:08:33.374713 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:33.374681    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp"]
Jul 29 23:08:33.402059 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:33.402024    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:08:34.681408 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:34.681376    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:08:38.936060 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:38.936001    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:08:38.967900 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:38.967872    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv"]
Jul 29 23:08:39.034969 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:39.034914    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:08:40.560014 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:40.559918    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp" podUID="0dc3135d-b991-4cf7-8472-6ae44e6c90e0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:08:44.726831 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:44.726770    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv" podUID="4982a45e-0edb-4713-ab54-ea4526f41fd8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:08:48.393671 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:48.393619    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp" podUID="0dc3135d-b991-4cf7-8472-6ae44e6c90e0" containerName="main" containerID="cri-o://fb1690bb2d83059bb04e3be15a344d70467ce8a2281a8b9a0e8a50a3f6424068" gracePeriod=45
Jul 29 23:08:48.644099 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:48.644048    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp"
Jul 29 23:08:48.713518 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:48.713494    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc3135d-b991-4cf7-8472-6ae44e6c90e0-tls-certs\") pod \"0dc3135d-b991-4cf7-8472-6ae44e6c90e0\" (UID: \"0dc3135d-b991-4cf7-8472-6ae44e6c90e0\") "
Jul 29 23:08:48.713627 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:48.713546    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgqrn\" (UniqueName: \"kubernetes.io/projected/0dc3135d-b991-4cf7-8472-6ae44e6c90e0-kube-api-access-jgqrn\") pod \"0dc3135d-b991-4cf7-8472-6ae44e6c90e0\" (UID: \"0dc3135d-b991-4cf7-8472-6ae44e6c90e0\") "
Jul 29 23:08:48.715598 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:48.715570    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dc3135d-b991-4cf7-8472-6ae44e6c90e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0dc3135d-b991-4cf7-8472-6ae44e6c90e0" (UID: "0dc3135d-b991-4cf7-8472-6ae44e6c90e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:08:48.715696 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:48.715625    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc3135d-b991-4cf7-8472-6ae44e6c90e0-kube-api-access-jgqrn" (OuterVolumeSpecName: "kube-api-access-jgqrn") pod "0dc3135d-b991-4cf7-8472-6ae44e6c90e0" (UID: "0dc3135d-b991-4cf7-8472-6ae44e6c90e0"). InnerVolumeSpecName "kube-api-access-jgqrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:08:48.815054 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:48.815031    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc3135d-b991-4cf7-8472-6ae44e6c90e0-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:08:48.815054 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:48.815051    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgqrn\" (UniqueName: \"kubernetes.io/projected/0dc3135d-b991-4cf7-8472-6ae44e6c90e0-kube-api-access-jgqrn\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:08:49.481041 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:49.481011    2581 generic.go:356] "Generic (PLEG): container finished" podID="0dc3135d-b991-4cf7-8472-6ae44e6c90e0" containerID="fb1690bb2d83059bb04e3be15a344d70467ce8a2281a8b9a0e8a50a3f6424068" exitCode=0
Jul 29 23:08:49.481533 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:49.481052    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp"
Jul 29 23:08:49.481533 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:49.481105    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp" event={"ID":"0dc3135d-b991-4cf7-8472-6ae44e6c90e0","Type":"ContainerDied","Data":"fb1690bb2d83059bb04e3be15a344d70467ce8a2281a8b9a0e8a50a3f6424068"}
Jul 29 23:08:49.481533 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:49.481170    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp" event={"ID":"0dc3135d-b991-4cf7-8472-6ae44e6c90e0","Type":"ContainerDied","Data":"4b4236a02e11e070208daa74e41c633a480bd5a1ace451de2b5956f3a8166370"}
Jul 29 23:08:49.481533 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:49.481206    2581 scope.go:117] "RemoveContainer" containerID="fb1690bb2d83059bb04e3be15a344d70467ce8a2281a8b9a0e8a50a3f6424068"
Jul 29 23:08:49.490326 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:49.490309    2581 scope.go:117] "RemoveContainer" containerID="fb1690bb2d83059bb04e3be15a344d70467ce8a2281a8b9a0e8a50a3f6424068"
Jul 29 23:08:49.490585 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:08:49.490567    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1690bb2d83059bb04e3be15a344d70467ce8a2281a8b9a0e8a50a3f6424068\": container with ID starting with fb1690bb2d83059bb04e3be15a344d70467ce8a2281a8b9a0e8a50a3f6424068 not found: ID does not exist" containerID="fb1690bb2d83059bb04e3be15a344d70467ce8a2281a8b9a0e8a50a3f6424068"
Jul 29 23:08:49.493825 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:49.490933    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1690bb2d83059bb04e3be15a344d70467ce8a2281a8b9a0e8a50a3f6424068"} err="failed to get container status \"fb1690bb2d83059bb04e3be15a344d70467ce8a2281a8b9a0e8a50a3f6424068\": rpc error: code = NotFound desc = could not find container \"fb1690bb2d83059bb04e3be15a344d70467ce8a2281a8b9a0e8a50a3f6424068\": container with ID starting with fb1690bb2d83059bb04e3be15a344d70467ce8a2281a8b9a0e8a50a3f6424068 not found: ID does not exist"
Jul 29 23:08:49.502604 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:49.502581    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp"]
Jul 29 23:08:49.507633 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:49.507612    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-6664595b9b-q2wjp"]
Jul 29 23:08:49.952727 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:49.952699    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dc3135d-b991-4cf7-8472-6ae44e6c90e0" path="/var/lib/kubelet/pods/0dc3135d-b991-4cf7-8472-6ae44e6c90e0/volumes"
Jul 29 23:08:53.427772 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:53.427734    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:08:53.522896 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:53.522862    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:08:53.987005 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:53.986969    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv" podUID="4982a45e-0edb-4713-ab54-ea4526f41fd8" containerName="main" containerID="cri-o://65d8a8c484478989527c15778898d8ed3fa34d995057264c96261dadc261b859" gracePeriod=45
Jul 29 23:08:54.243318 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:54.243253    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv"
Jul 29 23:08:54.364367 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:54.364340    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp5hd\" (UniqueName: \"kubernetes.io/projected/4982a45e-0edb-4713-ab54-ea4526f41fd8-kube-api-access-bp5hd\") pod \"4982a45e-0edb-4713-ab54-ea4526f41fd8\" (UID: \"4982a45e-0edb-4713-ab54-ea4526f41fd8\") "
Jul 29 23:08:54.364539 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:54.364425    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4982a45e-0edb-4713-ab54-ea4526f41fd8-tls-certs\") pod \"4982a45e-0edb-4713-ab54-ea4526f41fd8\" (UID: \"4982a45e-0edb-4713-ab54-ea4526f41fd8\") "
Jul 29 23:08:54.366509 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:54.366478    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4982a45e-0edb-4713-ab54-ea4526f41fd8-kube-api-access-bp5hd" (OuterVolumeSpecName: "kube-api-access-bp5hd") pod "4982a45e-0edb-4713-ab54-ea4526f41fd8" (UID: "4982a45e-0edb-4713-ab54-ea4526f41fd8"). InnerVolumeSpecName "kube-api-access-bp5hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:08:54.366623 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:54.366501    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4982a45e-0edb-4713-ab54-ea4526f41fd8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4982a45e-0edb-4713-ab54-ea4526f41fd8" (UID: "4982a45e-0edb-4713-ab54-ea4526f41fd8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:08:54.464976 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:54.464947    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp5hd\" (UniqueName: \"kubernetes.io/projected/4982a45e-0edb-4713-ab54-ea4526f41fd8-kube-api-access-bp5hd\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:08:54.464976 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:54.464975    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4982a45e-0edb-4713-ab54-ea4526f41fd8-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:08:54.507302 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:54.507221    2581 generic.go:356] "Generic (PLEG): container finished" podID="4982a45e-0edb-4713-ab54-ea4526f41fd8" containerID="65d8a8c484478989527c15778898d8ed3fa34d995057264c96261dadc261b859" exitCode=0
Jul 29 23:08:54.507302 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:54.507267    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv"
Jul 29 23:08:54.507302 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:54.507279    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv" event={"ID":"4982a45e-0edb-4713-ab54-ea4526f41fd8","Type":"ContainerDied","Data":"65d8a8c484478989527c15778898d8ed3fa34d995057264c96261dadc261b859"}
Jul 29 23:08:54.507549 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:54.507324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv" event={"ID":"4982a45e-0edb-4713-ab54-ea4526f41fd8","Type":"ContainerDied","Data":"3ceb20fd9a2b5b596780db81232e20998cdeaab86edd7fb6ca6a95f7d8032f53"}
Jul 29 23:08:54.507549 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:54.507345    2581 scope.go:117] "RemoveContainer" containerID="65d8a8c484478989527c15778898d8ed3fa34d995057264c96261dadc261b859"
Jul 29 23:08:54.516619 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:54.516600    2581 scope.go:117] "RemoveContainer" containerID="65d8a8c484478989527c15778898d8ed3fa34d995057264c96261dadc261b859"
Jul 29 23:08:54.516950 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:08:54.516927    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d8a8c484478989527c15778898d8ed3fa34d995057264c96261dadc261b859\": container with ID starting with 65d8a8c484478989527c15778898d8ed3fa34d995057264c96261dadc261b859 not found: ID does not exist" containerID="65d8a8c484478989527c15778898d8ed3fa34d995057264c96261dadc261b859"
Jul 29 23:08:54.516996 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:54.516959    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d8a8c484478989527c15778898d8ed3fa34d995057264c96261dadc261b859"} err="failed to get container status \"65d8a8c484478989527c15778898d8ed3fa34d995057264c96261dadc261b859\": rpc error: code = NotFound desc = could not find container \"65d8a8c484478989527c15778898d8ed3fa34d995057264c96261dadc261b859\": container with ID starting with 65d8a8c484478989527c15778898d8ed3fa34d995057264c96261dadc261b859 not found: ID does not exist"
Jul 29 23:08:54.529572 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:54.529549    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv"]
Jul 29 23:08:54.533345 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:54.533320    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheg4zbv"]
Jul 29 23:08:55.953408 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:55.953378    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4982a45e-0edb-4713-ab54-ea4526f41fd8" path="/var/lib/kubelet/pods/4982a45e-0edb-4713-ab54-ea4526f41fd8/volumes"
Jul 29 23:08:59.245988 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:08:59.245953    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:09:04.703643 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:04.703610    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:09:04.803651 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:04.803621    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:09:05.741205 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:05.741165    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:09:05.898452 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:05.898422    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp"]
Jul 29 23:09:05.898836 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:05.898822    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dc3135d-b991-4cf7-8472-6ae44e6c90e0" containerName="main"
Jul 29 23:09:05.898889 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:05.898837    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dc3135d-b991-4cf7-8472-6ae44e6c90e0" containerName="main"
Jul 29 23:09:05.898889 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:05.898849    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4982a45e-0edb-4713-ab54-ea4526f41fd8" containerName="main"
Jul 29 23:09:05.898889 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:05.898856    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4982a45e-0edb-4713-ab54-ea4526f41fd8" containerName="main"
Jul 29 23:09:05.898982 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:05.898915    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dc3135d-b991-4cf7-8472-6ae44e6c90e0" containerName="main"
Jul 29 23:09:05.898982 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:05.898927    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4982a45e-0edb-4713-ab54-ea4526f41fd8" containerName="main"
Jul 29 23:09:05.909129 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:05.909107    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp"
Jul 29 23:09:05.911146 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:05.911121    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp"]
Jul 29 23:09:05.913444 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:05.913423    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 29 23:09:05.913444 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:05.913434    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Jul 29 23:09:05.913617 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:05.913459    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Jul 29 23:09:05.913617 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:05.913484    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-vdht2\""
Jul 29 23:09:05.913617 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:05.913436    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-cqglg\""
Jul 29 23:09:05.937810 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:05.937771    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:09:06.055758 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:06.055687    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqqkk\" (UniqueName: \"kubernetes.io/projected/35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3-kube-api-access-qqqkk\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp\" (UID: \"35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp"
Jul 29 23:09:06.055935 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:06.055765    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp\" (UID: \"35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp"
Jul 29 23:09:06.156410 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:06.156382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp\" (UID: \"35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp"
Jul 29 23:09:06.156571 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:06.156550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqkk\" (UniqueName: \"kubernetes.io/projected/35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3-kube-api-access-qqqkk\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp\" (UID: \"35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp"
Jul 29 23:09:06.158600 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:06.158580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp\" (UID: \"35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp"
Jul 29 23:09:06.164062 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:06.164043    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqqkk\" (UniqueName: \"kubernetes.io/projected/35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3-kube-api-access-qqqkk\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp\" (UID: \"35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp"
Jul 29 23:09:06.219886 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:06.219864    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp"
Jul 29 23:09:06.346932 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:06.346843    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp"]
Jul 29 23:09:06.349130 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:09:06.349105    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a0ad3d_d41f_4d29_851d_d9fcf77c0ca3.slice/crio-d0fe9c115353fb1d0e68fd73f57e5db01bf1f92f3712e9ffde34d686956acf76 WatchSource:0}: Error finding container d0fe9c115353fb1d0e68fd73f57e5db01bf1f92f3712e9ffde34d686956acf76: Status 404 returned error can't find the container with id d0fe9c115353fb1d0e68fd73f57e5db01bf1f92f3712e9ffde34d686956acf76
Jul 29 23:09:06.564985 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:06.564949    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp" event={"ID":"35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3","Type":"ContainerStarted","Data":"a96bb843262cfed9c55ebe3503664deaedb843570b80ee992a0408f94b32940a"}
Jul 29 23:09:06.565152 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:06.565004    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp" event={"ID":"35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3","Type":"ContainerStarted","Data":"d0fe9c115353fb1d0e68fd73f57e5db01bf1f92f3712e9ffde34d686956acf76"}
Jul 29 23:09:06.565152 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:06.565044    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp"
Jul 29 23:09:13.831212 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:13.831176    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:09:13.926532 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:13.926504    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:09:14.522892 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:14.522859    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:09:15.521227 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:15.521169    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:09:19.923090 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:19.923056    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:09:20.051926 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.051890    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:09:20.097608 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.097559    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp" podStartSLOduration=15.097543801 podStartE2EDuration="15.097543801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:09:06.582691616 +0000 UTC m=+1369.141036431" watchObservedRunningTime="2026-07-29 23:09:20.097543801 +0000 UTC m=+1382.655888617"
Jul 29 23:09:20.098853 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.098835    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z"]
Jul 29 23:09:20.103416 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.103401    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z"
Jul 29 23:09:20.106590 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.106569    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Jul 29 23:09:20.108119 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.108092    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-jzxzj\""
Jul 29 23:09:20.108119 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.108092    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-dmg8k\""
Jul 29 23:09:20.108258 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.108140    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Jul 29 23:09:20.108258 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.108114    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 29 23:09:20.111924 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.111904    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z"]
Jul 29 23:09:20.150187 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.150169    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:09:20.175284 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.175230    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a610336a-77e0-4255-b15d-ac9c3896a88a-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z\" (UID: \"a610336a-77e0-4255-b15d-ac9c3896a88a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z"
Jul 29 23:09:20.175371 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.175297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz4zk\" (UniqueName: \"kubernetes.io/projected/a610336a-77e0-4255-b15d-ac9c3896a88a-kube-api-access-jz4zk\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z\" (UID: \"a610336a-77e0-4255-b15d-ac9c3896a88a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z"
Jul 29 23:09:20.276515 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.276487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4zk\" (UniqueName: \"kubernetes.io/projected/a610336a-77e0-4255-b15d-ac9c3896a88a-kube-api-access-jz4zk\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z\" (UID: \"a610336a-77e0-4255-b15d-ac9c3896a88a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z"
Jul 29 23:09:20.276627 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.276555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a610336a-77e0-4255-b15d-ac9c3896a88a-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z\" (UID: \"a610336a-77e0-4255-b15d-ac9c3896a88a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z"
Jul 29 23:09:20.278944 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.278923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a610336a-77e0-4255-b15d-ac9c3896a88a-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z\" (UID: \"a610336a-77e0-4255-b15d-ac9c3896a88a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z"
Jul 29 23:09:20.284593 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.284567    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4zk\" (UniqueName: \"kubernetes.io/projected/a610336a-77e0-4255-b15d-ac9c3896a88a-kube-api-access-jz4zk\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z\" (UID: \"a610336a-77e0-4255-b15d-ac9c3896a88a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z"
Jul 29 23:09:20.414232 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.414201    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z"
Jul 29 23:09:20.546000 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.545974    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z"]
Jul 29 23:09:20.548378 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:09:20.548349    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda610336a_77e0_4255_b15d_ac9c3896a88a.slice/crio-6ee1b8890cb869293f15fffeef9ddd920d8ad1a499e7a2c152d744bfe5553b11 WatchSource:0}: Error finding container 6ee1b8890cb869293f15fffeef9ddd920d8ad1a499e7a2c152d744bfe5553b11: Status 404 returned error can't find the container with id 6ee1b8890cb869293f15fffeef9ddd920d8ad1a499e7a2c152d744bfe5553b11
Jul 29 23:09:20.630962 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.630933    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z" event={"ID":"a610336a-77e0-4255-b15d-ac9c3896a88a","Type":"ContainerStarted","Data":"0a805af547e1aa18b52b530cd77319304472fefe9ad2432ce65b90ef12000a43"}
Jul 29 23:09:20.631103 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.631025    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z"
Jul 29 23:09:20.631103 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.631053    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z" event={"ID":"a610336a-77e0-4255-b15d-ac9c3896a88a","Type":"ContainerStarted","Data":"6ee1b8890cb869293f15fffeef9ddd920d8ad1a499e7a2c152d744bfe5553b11"}
Jul 29 23:09:20.647489 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:20.647444    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z" podStartSLOduration=0.647426951 podStartE2EDuration="647.426951ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:09:20.646528607 +0000 UTC m=+1383.204873426" watchObservedRunningTime="2026-07-29 23:09:20.647426951 +0000 UTC m=+1383.205771766"
Jul 29 23:09:25.876338 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:25.876302    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:09:25.976643 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:25.976592    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:09:37.574012 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:37.573982    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp"
Jul 29 23:09:42.861618 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:42.861580    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:09:51.643457 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:51.643430    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z"
Jul 29 23:09:52.802783 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:52.802737    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:09:52.808988 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:52.808960    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z"]
Jul 29 23:09:52.868157 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:09:52.868126    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:10:01.643627 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:01.643583    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z" podUID="a610336a-77e0-4255-b15d-ac9c3896a88a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:10:07.829272 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:07.829210    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z" podUID="a610336a-77e0-4255-b15d-ac9c3896a88a" containerName="main" containerID="cri-o://0a805af547e1aa18b52b530cd77319304472fefe9ad2432ce65b90ef12000a43" gracePeriod=45
Jul 29 23:10:08.069439 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:08.069416    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z"
Jul 29 23:10:08.258758 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:08.258728    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a610336a-77e0-4255-b15d-ac9c3896a88a-tls-certs\") pod \"a610336a-77e0-4255-b15d-ac9c3896a88a\" (UID: \"a610336a-77e0-4255-b15d-ac9c3896a88a\") "
Jul 29 23:10:08.258972 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:08.258821    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz4zk\" (UniqueName: \"kubernetes.io/projected/a610336a-77e0-4255-b15d-ac9c3896a88a-kube-api-access-jz4zk\") pod \"a610336a-77e0-4255-b15d-ac9c3896a88a\" (UID: \"a610336a-77e0-4255-b15d-ac9c3896a88a\") "
Jul 29 23:10:08.261005 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:08.260973    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a610336a-77e0-4255-b15d-ac9c3896a88a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a610336a-77e0-4255-b15d-ac9c3896a88a" (UID: "a610336a-77e0-4255-b15d-ac9c3896a88a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:10:08.261113 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:08.260979    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a610336a-77e0-4255-b15d-ac9c3896a88a-kube-api-access-jz4zk" (OuterVolumeSpecName: "kube-api-access-jz4zk") pod "a610336a-77e0-4255-b15d-ac9c3896a88a" (UID: "a610336a-77e0-4255-b15d-ac9c3896a88a"). InnerVolumeSpecName "kube-api-access-jz4zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:10:08.360016 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:08.359989    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a610336a-77e0-4255-b15d-ac9c3896a88a-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:10:08.360016 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:08.360014    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz4zk\" (UniqueName: \"kubernetes.io/projected/a610336a-77e0-4255-b15d-ac9c3896a88a-kube-api-access-jz4zk\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:10:08.864299 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:08.864265    2581 generic.go:356] "Generic (PLEG): container finished" podID="a610336a-77e0-4255-b15d-ac9c3896a88a" containerID="0a805af547e1aa18b52b530cd77319304472fefe9ad2432ce65b90ef12000a43" exitCode=0
Jul 29 23:10:08.864792 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:08.864301    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z" event={"ID":"a610336a-77e0-4255-b15d-ac9c3896a88a","Type":"ContainerDied","Data":"0a805af547e1aa18b52b530cd77319304472fefe9ad2432ce65b90ef12000a43"}
Jul 29 23:10:08.864792 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:08.864320    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z"
Jul 29 23:10:08.864792 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:08.864367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z" event={"ID":"a610336a-77e0-4255-b15d-ac9c3896a88a","Type":"ContainerDied","Data":"6ee1b8890cb869293f15fffeef9ddd920d8ad1a499e7a2c152d744bfe5553b11"}
Jul 29 23:10:08.864792 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:08.864398    2581 scope.go:117] "RemoveContainer" containerID="0a805af547e1aa18b52b530cd77319304472fefe9ad2432ce65b90ef12000a43"
Jul 29 23:10:08.873550 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:08.873532    2581 scope.go:117] "RemoveContainer" containerID="0a805af547e1aa18b52b530cd77319304472fefe9ad2432ce65b90ef12000a43"
Jul 29 23:10:08.873791 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:10:08.873772    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a805af547e1aa18b52b530cd77319304472fefe9ad2432ce65b90ef12000a43\": container with ID starting with 0a805af547e1aa18b52b530cd77319304472fefe9ad2432ce65b90ef12000a43 not found: ID does not exist" containerID="0a805af547e1aa18b52b530cd77319304472fefe9ad2432ce65b90ef12000a43"
Jul 29 23:10:08.873866 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:08.873815    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a805af547e1aa18b52b530cd77319304472fefe9ad2432ce65b90ef12000a43"} err="failed to get container status \"0a805af547e1aa18b52b530cd77319304472fefe9ad2432ce65b90ef12000a43\": rpc error: code = NotFound desc = could not find container \"0a805af547e1aa18b52b530cd77319304472fefe9ad2432ce65b90ef12000a43\": container with ID starting with 0a805af547e1aa18b52b530cd77319304472fefe9ad2432ce65b90ef12000a43 not found: ID does not exist"
Jul 29 23:10:08.886566 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:08.886546    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z"]
Jul 29 23:10:08.890160 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:08.890138    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557wl24z"]
Jul 29 23:10:09.952288 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:09.952215    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a610336a-77e0-4255-b15d-ac9c3896a88a" path="/var/lib/kubelet/pods/a610336a-77e0-4255-b15d-ac9c3896a88a/volumes"
Jul 29 23:10:12.864516 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:12.864482    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:10:12.970940 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:12.970908    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:10:17.431617 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.431584    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:10:17.597333 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.597302    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb"]
Jul 29 23:10:17.597683 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.597669    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a610336a-77e0-4255-b15d-ac9c3896a88a" containerName="main"
Jul 29 23:10:17.597740 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.597684    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a610336a-77e0-4255-b15d-ac9c3896a88a" containerName="main"
Jul 29 23:10:17.597775 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.597770    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a610336a-77e0-4255-b15d-ac9c3896a88a" containerName="main"
Jul 29 23:10:17.600642 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.600626    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb"
Jul 29 23:10:17.604903 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.604878    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Jul 29 23:10:17.604903 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.604892    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-sdgp8\""
Jul 29 23:10:17.605294 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.604884    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Jul 29 23:10:17.605294 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.604964    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-6cz5w\""
Jul 29 23:10:17.605294 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.605037    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 29 23:10:17.610388 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.610369    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb"]
Jul 29 23:10:17.625069 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.625048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vck4v\" (UniqueName: \"kubernetes.io/projected/0136556a-9085-4aeb-b338-31a677e0e2fa-kube-api-access-vck4v\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb\" (UID: \"0136556a-9085-4aeb-b338-31a677e0e2fa\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb"
Jul 29 23:10:17.625161 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.625091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0136556a-9085-4aeb-b338-31a677e0e2fa-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb\" (UID: \"0136556a-9085-4aeb-b338-31a677e0e2fa\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb"
Jul 29 23:10:17.634357 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.634315    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:10:17.726261 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.726200    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vck4v\" (UniqueName: \"kubernetes.io/projected/0136556a-9085-4aeb-b338-31a677e0e2fa-kube-api-access-vck4v\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb\" (UID: \"0136556a-9085-4aeb-b338-31a677e0e2fa\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb"
Jul 29 23:10:17.726261 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.726243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0136556a-9085-4aeb-b338-31a677e0e2fa-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb\" (UID: \"0136556a-9085-4aeb-b338-31a677e0e2fa\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb"
Jul 29 23:10:17.728402 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.728378    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0136556a-9085-4aeb-b338-31a677e0e2fa-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb\" (UID: \"0136556a-9085-4aeb-b338-31a677e0e2fa\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb"
Jul 29 23:10:17.735150 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.735115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vck4v\" (UniqueName: \"kubernetes.io/projected/0136556a-9085-4aeb-b338-31a677e0e2fa-kube-api-access-vck4v\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb\" (UID: \"0136556a-9085-4aeb-b338-31a677e0e2fa\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb"
Jul 29 23:10:17.914400 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.914371    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-sdgp8\""
Jul 29 23:10:17.925159 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.925140    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-6cz5w\""
Jul 29 23:10:17.933402 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:17.933381    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb"
Jul 29 23:10:18.067327 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:18.067304    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb"]
Jul 29 23:10:18.069979 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:10:18.069945    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0136556a_9085_4aeb_b338_31a677e0e2fa.slice/crio-382d5ef8086f783e05bc2dc67f806770ea96a2cc967b6f33d5843b2960432152 WatchSource:0}: Error finding container 382d5ef8086f783e05bc2dc67f806770ea96a2cc967b6f33d5843b2960432152: Status 404 returned error can't find the container with id 382d5ef8086f783e05bc2dc67f806770ea96a2cc967b6f33d5843b2960432152
Jul 29 23:10:18.916291 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:18.916255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb" event={"ID":"0136556a-9085-4aeb-b338-31a677e0e2fa","Type":"ContainerStarted","Data":"29c8250ae99c72b8245a735084327443ad37f729f8a573bdec25da1b4dc33d3a"}
Jul 29 23:10:18.916712 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:18.916330    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb" event={"ID":"0136556a-9085-4aeb-b338-31a677e0e2fa","Type":"ContainerStarted","Data":"382d5ef8086f783e05bc2dc67f806770ea96a2cc967b6f33d5843b2960432152"}
Jul 29 23:10:18.916712 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:18.916403    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb"
Jul 29 23:10:18.933848 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:18.933779    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb" podStartSLOduration=1.933763746 podStartE2EDuration="1.933763746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:10:18.931832736 +0000 UTC m=+1441.490177617" watchObservedRunningTime="2026-07-29 23:10:18.933763746 +0000 UTC m=+1441.492108569"
Jul 29 23:10:19.527349 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:19.527317    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:10:25.036686 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:25.036572    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:10:25.137327 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:25.137297    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:10:28.915530 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:28.915487    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:10:49.923910 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:49.923880    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb"
Jul 29 23:10:51.567923 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:51.567892    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb"]
Jul 29 23:10:51.570121 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:51.570083    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:10:51.665970 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:51.665937    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:10:59.923824 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:10:59.923768    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb" podUID="0136556a-9085-4aeb-b338-31a677e0e2fa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:11:06.587862 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:06.587824    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb" podUID="0136556a-9085-4aeb-b338-31a677e0e2fa" containerName="main" containerID="cri-o://29c8250ae99c72b8245a735084327443ad37f729f8a573bdec25da1b4dc33d3a" gracePeriod=45
Jul 29 23:11:06.837642 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:06.837619    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb"
Jul 29 23:11:06.907411 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:06.907346    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0136556a-9085-4aeb-b338-31a677e0e2fa-tls-certs\") pod \"0136556a-9085-4aeb-b338-31a677e0e2fa\" (UID: \"0136556a-9085-4aeb-b338-31a677e0e2fa\") "
Jul 29 23:11:06.907523 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:06.907438    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vck4v\" (UniqueName: \"kubernetes.io/projected/0136556a-9085-4aeb-b338-31a677e0e2fa-kube-api-access-vck4v\") pod \"0136556a-9085-4aeb-b338-31a677e0e2fa\" (UID: \"0136556a-9085-4aeb-b338-31a677e0e2fa\") "
Jul 29 23:11:06.909466 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:06.909434    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0136556a-9085-4aeb-b338-31a677e0e2fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0136556a-9085-4aeb-b338-31a677e0e2fa" (UID: "0136556a-9085-4aeb-b338-31a677e0e2fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:11:06.909578 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:06.909501    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0136556a-9085-4aeb-b338-31a677e0e2fa-kube-api-access-vck4v" (OuterVolumeSpecName: "kube-api-access-vck4v") pod "0136556a-9085-4aeb-b338-31a677e0e2fa" (UID: "0136556a-9085-4aeb-b338-31a677e0e2fa"). InnerVolumeSpecName "kube-api-access-vck4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:11:07.008287 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:07.008264    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0136556a-9085-4aeb-b338-31a677e0e2fa-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:11:07.008287 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:07.008286    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vck4v\" (UniqueName: \"kubernetes.io/projected/0136556a-9085-4aeb-b338-31a677e0e2fa-kube-api-access-vck4v\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:11:07.142243 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:07.142212    2581 generic.go:356] "Generic (PLEG): container finished" podID="0136556a-9085-4aeb-b338-31a677e0e2fa" containerID="29c8250ae99c72b8245a735084327443ad37f729f8a573bdec25da1b4dc33d3a" exitCode=0
Jul 29 23:11:07.142355 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:07.142252    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb"
Jul 29 23:11:07.142355 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:07.142261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb" event={"ID":"0136556a-9085-4aeb-b338-31a677e0e2fa","Type":"ContainerDied","Data":"29c8250ae99c72b8245a735084327443ad37f729f8a573bdec25da1b4dc33d3a"}
Jul 29 23:11:07.142355 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:07.142311    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb" event={"ID":"0136556a-9085-4aeb-b338-31a677e0e2fa","Type":"ContainerDied","Data":"382d5ef8086f783e05bc2dc67f806770ea96a2cc967b6f33d5843b2960432152"}
Jul 29 23:11:07.142355 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:07.142334    2581 scope.go:117] "RemoveContainer" containerID="29c8250ae99c72b8245a735084327443ad37f729f8a573bdec25da1b4dc33d3a"
Jul 29 23:11:07.151022 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:07.151007    2581 scope.go:117] "RemoveContainer" containerID="29c8250ae99c72b8245a735084327443ad37f729f8a573bdec25da1b4dc33d3a"
Jul 29 23:11:07.151273 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:11:07.151252    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29c8250ae99c72b8245a735084327443ad37f729f8a573bdec25da1b4dc33d3a\": container with ID starting with 29c8250ae99c72b8245a735084327443ad37f729f8a573bdec25da1b4dc33d3a not found: ID does not exist" containerID="29c8250ae99c72b8245a735084327443ad37f729f8a573bdec25da1b4dc33d3a"
Jul 29 23:11:07.151325 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:07.151281    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29c8250ae99c72b8245a735084327443ad37f729f8a573bdec25da1b4dc33d3a"} err="failed to get container status \"29c8250ae99c72b8245a735084327443ad37f729f8a573bdec25da1b4dc33d3a\": rpc error: code = NotFound desc = could not find container \"29c8250ae99c72b8245a735084327443ad37f729f8a573bdec25da1b4dc33d3a\": container with ID starting with 29c8250ae99c72b8245a735084327443ad37f729f8a573bdec25da1b4dc33d3a not found: ID does not exist"
Jul 29 23:11:07.163472 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:07.163423    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb"]
Jul 29 23:11:07.166184 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:07.166163    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2kpgb"]
Jul 29 23:11:07.602515 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:07.602484    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp"]
Jul 29 23:11:07.605654 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:07.605590    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:11:07.668394 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:07.668368    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:11:07.953422 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:07.953346    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0136556a-9085-4aeb-b338-31a677e0e2fa" path="/var/lib/kubelet/pods/0136556a-9085-4aeb-b338-31a677e0e2fa/volumes"
Jul 29 23:11:11.573179 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:11.573134    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:11:11.667158 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:11.667131    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:11:17.506102 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:17.506070    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:11:17.574584 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:17.574552    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp" podUID="35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:11:22.627136 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:22.627095    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp" podUID="35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3" containerName="main" containerID="cri-o://a96bb843262cfed9c55ebe3503664deaedb843570b80ee992a0408f94b32940a" gracePeriod=45
Jul 29 23:11:22.874625 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:22.874603    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp"
Jul 29 23:11:23.030087 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:23.030012    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:11:23.034058 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:23.034040    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqqkk\" (UniqueName: \"kubernetes.io/projected/35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3-kube-api-access-qqqkk\") pod \"35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3\" (UID: \"35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3\") "
Jul 29 23:11:23.034126 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:23.034103    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3-tls-certs\") pod \"35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3\" (UID: \"35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3\") "
Jul 29 23:11:23.036257 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:23.036224    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3-kube-api-access-qqqkk" (OuterVolumeSpecName: "kube-api-access-qqqkk") pod "35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3" (UID: "35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3"). InnerVolumeSpecName "kube-api-access-qqqkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:11:23.036257 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:23.036248    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3" (UID: "35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:11:23.112895 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:23.112862    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:11:23.134578 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:23.134555    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqqkk\" (UniqueName: \"kubernetes.io/projected/35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3-kube-api-access-qqqkk\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:11:23.134578 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:23.134576    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:11:23.219063 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:23.219036    2581 generic.go:356] "Generic (PLEG): container finished" podID="35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3" containerID="a96bb843262cfed9c55ebe3503664deaedb843570b80ee992a0408f94b32940a" exitCode=0
Jul 29 23:11:23.219187 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:23.219066    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp" event={"ID":"35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3","Type":"ContainerDied","Data":"a96bb843262cfed9c55ebe3503664deaedb843570b80ee992a0408f94b32940a"}
Jul 29 23:11:23.219187 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:23.219086    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp"
Jul 29 23:11:23.219187 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:23.219126    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp" event={"ID":"35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3","Type":"ContainerDied","Data":"d0fe9c115353fb1d0e68fd73f57e5db01bf1f92f3712e9ffde34d686956acf76"}
Jul 29 23:11:23.219187 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:23.219149    2581 scope.go:117] "RemoveContainer" containerID="a96bb843262cfed9c55ebe3503664deaedb843570b80ee992a0408f94b32940a"
Jul 29 23:11:23.228073 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:23.228057    2581 scope.go:117] "RemoveContainer" containerID="a96bb843262cfed9c55ebe3503664deaedb843570b80ee992a0408f94b32940a"
Jul 29 23:11:23.228327 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:11:23.228307    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a96bb843262cfed9c55ebe3503664deaedb843570b80ee992a0408f94b32940a\": container with ID starting with a96bb843262cfed9c55ebe3503664deaedb843570b80ee992a0408f94b32940a not found: ID does not exist" containerID="a96bb843262cfed9c55ebe3503664deaedb843570b80ee992a0408f94b32940a"
Jul 29 23:11:23.228371 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:23.228336    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96bb843262cfed9c55ebe3503664deaedb843570b80ee992a0408f94b32940a"} err="failed to get container status \"a96bb843262cfed9c55ebe3503664deaedb843570b80ee992a0408f94b32940a\": rpc error: code = NotFound desc = could not find container \"a96bb843262cfed9c55ebe3503664deaedb843570b80ee992a0408f94b32940a\": container with ID starting with a96bb843262cfed9c55ebe3503664deaedb843570b80ee992a0408f94b32940a not found: ID does not exist"
Jul 29 23:11:23.242756 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:23.242736    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp"]
Jul 29 23:11:23.246649 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:23.246628    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6c898f97bcnt7zp"]
Jul 29 23:11:23.951776 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:23.951747    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3" path="/var/lib/kubelet/pods/35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3/volumes"
Jul 29 23:11:27.463578 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:27.463550    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:11:27.562998 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:27.562965    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:11:28.417152 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.417118    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:11:28.674304 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.674231    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59"]
Jul 29 23:11:28.674632 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.674615    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0136556a-9085-4aeb-b338-31a677e0e2fa" containerName="main"
Jul 29 23:11:28.674632 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.674628    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0136556a-9085-4aeb-b338-31a677e0e2fa" containerName="main"
Jul 29 23:11:28.674695 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.674643    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3" containerName="main"
Jul 29 23:11:28.674695 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.674648    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3" containerName="main"
Jul 29 23:11:28.674771 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.674717    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0136556a-9085-4aeb-b338-31a677e0e2fa" containerName="main"
Jul 29 23:11:28.674771 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.674727    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a0ad3d-d41f-4d29-851d-d9fcf77c0ca3" containerName="main"
Jul 29 23:11:28.679207 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.679191    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59"
Jul 29 23:11:28.683519 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.683495    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-rwhd7\""
Jul 29 23:11:28.683661 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.683523    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-ds4mk\""
Jul 29 23:11:28.683661 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.683536    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Jul 29 23:11:28.683661 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.683554    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 29 23:11:28.683661 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.683584    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Jul 29 23:11:28.686626 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.686606    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59"]
Jul 29 23:11:28.709844 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.709825    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:11:28.786179 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.786150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmms8\" (UniqueName: \"kubernetes.io/projected/586ff4c9-ea54-4a54-8289-4add8b20d810-kube-api-access-qmms8\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59\" (UID: \"586ff4c9-ea54-4a54-8289-4add8b20d810\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59"
Jul 29 23:11:28.786327 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.786190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586ff4c9-ea54-4a54-8289-4add8b20d810-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59\" (UID: \"586ff4c9-ea54-4a54-8289-4add8b20d810\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59"
Jul 29 23:11:28.887592 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.887562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmms8\" (UniqueName: \"kubernetes.io/projected/586ff4c9-ea54-4a54-8289-4add8b20d810-kube-api-access-qmms8\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59\" (UID: \"586ff4c9-ea54-4a54-8289-4add8b20d810\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59"
Jul 29 23:11:28.887693 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.887602    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586ff4c9-ea54-4a54-8289-4add8b20d810-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59\" (UID: \"586ff4c9-ea54-4a54-8289-4add8b20d810\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59"
Jul 29 23:11:28.890121 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.890098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586ff4c9-ea54-4a54-8289-4add8b20d810-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59\" (UID: \"586ff4c9-ea54-4a54-8289-4add8b20d810\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59"
Jul 29 23:11:28.897004 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.896983    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmms8\" (UniqueName: \"kubernetes.io/projected/586ff4c9-ea54-4a54-8289-4add8b20d810-kube-api-access-qmms8\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59\" (UID: \"586ff4c9-ea54-4a54-8289-4add8b20d810\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59"
Jul 29 23:11:28.990210 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:28.990153    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59"
Jul 29 23:11:29.121331 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:29.121298    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59"]
Jul 29 23:11:29.124970 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:11:29.124946    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586ff4c9_ea54_4a54_8289_4add8b20d810.slice/crio-d472f8f8c8c15ea028fed89a095f353735bc7c0ed1bca0ecc8f27fbe58504881 WatchSource:0}: Error finding container d472f8f8c8c15ea028fed89a095f353735bc7c0ed1bca0ecc8f27fbe58504881: Status 404 returned error can't find the container with id d472f8f8c8c15ea028fed89a095f353735bc7c0ed1bca0ecc8f27fbe58504881
Jul 29 23:11:29.248702 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:29.248613    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59" event={"ID":"586ff4c9-ea54-4a54-8289-4add8b20d810","Type":"ContainerStarted","Data":"eced9b9b5ba05d21ded30b15551f00ffe8068ffa6f37766863df428f175afe6d"}
Jul 29 23:11:29.248702 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:29.248682    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59"
Jul 29 23:11:29.248912 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:29.248708    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59" event={"ID":"586ff4c9-ea54-4a54-8289-4add8b20d810","Type":"ContainerStarted","Data":"d472f8f8c8c15ea028fed89a095f353735bc7c0ed1bca0ecc8f27fbe58504881"}
Jul 29 23:11:33.801306 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:33.801273    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:11:39.554689 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:39.554614    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:11:39.625580 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:39.625542    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:11:43.237940 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.237904    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:11:43.511342 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.511230    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59" podStartSLOduration=15.511210051 podStartE2EDuration="15.511210051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:11:29.271760668 +0000 UTC m=+1511.830105485" watchObservedRunningTime="2026-07-29 23:11:43.511210051 +0000 UTC m=+1526.069554868"
Jul 29 23:11:43.511600 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.511579    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb"]
Jul 29 23:11:43.515064 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.515045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb"
Jul 29 23:11:43.519175 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.519120    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-4qvt7\""
Jul 29 23:11:43.519175 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.519173    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-ndsv9\""
Jul 29 23:11:43.519432 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.519214    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Jul 29 23:11:43.519529 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.519465    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 29 23:11:43.519588 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.519557    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Jul 29 23:11:43.524739 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.524676    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb"]
Jul 29 23:11:43.554141 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.554107    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:11:43.612542 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.612506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgvql\" (UniqueName: \"kubernetes.io/projected/46e96b4d-271c-4448-954c-70ed1e9a70ff-kube-api-access-lgvql\") pod \"auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb\" (UID: \"46e96b4d-271c-4448-954c-70ed1e9a70ff\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb"
Jul 29 23:11:43.612728 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.612578    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e96b4d-271c-4448-954c-70ed1e9a70ff-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb\" (UID: \"46e96b4d-271c-4448-954c-70ed1e9a70ff\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb"
Jul 29 23:11:43.714009 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.713976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgvql\" (UniqueName: \"kubernetes.io/projected/46e96b4d-271c-4448-954c-70ed1e9a70ff-kube-api-access-lgvql\") pod \"auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb\" (UID: \"46e96b4d-271c-4448-954c-70ed1e9a70ff\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb"
Jul 29 23:11:43.714200 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.714052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e96b4d-271c-4448-954c-70ed1e9a70ff-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb\" (UID: \"46e96b4d-271c-4448-954c-70ed1e9a70ff\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb"
Jul 29 23:11:43.716514 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.716490    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e96b4d-271c-4448-954c-70ed1e9a70ff-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb\" (UID: \"46e96b4d-271c-4448-954c-70ed1e9a70ff\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb"
Jul 29 23:11:43.722916 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.722892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgvql\" (UniqueName: \"kubernetes.io/projected/46e96b4d-271c-4448-954c-70ed1e9a70ff-kube-api-access-lgvql\") pod \"auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb\" (UID: \"46e96b4d-271c-4448-954c-70ed1e9a70ff\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb"
Jul 29 23:11:43.828020 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.827935    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb"
Jul 29 23:11:43.960674 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:43.960650    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb"]
Jul 29 23:11:43.963005 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:11:43.962979    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e96b4d_271c_4448_954c_70ed1e9a70ff.slice/crio-872ce6cf2aa793256578c0951764596837c32a8faed67e3fa68d1fd7fe3eada5 WatchSource:0}: Error finding container 872ce6cf2aa793256578c0951764596837c32a8faed67e3fa68d1fd7fe3eada5: Status 404 returned error can't find the container with id 872ce6cf2aa793256578c0951764596837c32a8faed67e3fa68d1fd7fe3eada5
Jul 29 23:11:44.319056 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:44.319020    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb" event={"ID":"46e96b4d-271c-4448-954c-70ed1e9a70ff","Type":"ContainerStarted","Data":"0724688b48324f503ec95cfd523a89df514b7a2d4163cd430cc57dfae8ee65e5"}
Jul 29 23:11:44.319506 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:44.319079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb" event={"ID":"46e96b4d-271c-4448-954c-70ed1e9a70ff","Type":"ContainerStarted","Data":"872ce6cf2aa793256578c0951764596837c32a8faed67e3fa68d1fd7fe3eada5"}
Jul 29 23:11:44.319506 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:44.319142    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb"
Jul 29 23:11:44.338400 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:44.338350    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb" podStartSLOduration=1.338332194 podStartE2EDuration="1.338332194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:11:44.336042227 +0000 UTC m=+1526.894387045" watchObservedRunningTime="2026-07-29 23:11:44.338332194 +0000 UTC m=+1526.896677011"
Jul 29 23:11:49.887337 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:49.887304    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:11:59.900113 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:11:59.900076    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:12:00.254991 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:00.254915    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59"
Jul 29 23:12:03.032260 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:03.032226    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:12:03.035964 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:03.035936    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59"]
Jul 29 23:12:03.100047 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:03.100017    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:12:10.255498 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:10.255460    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59" podUID="586ff4c9-ea54-4a54-8289-4add8b20d810" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:12:15.326241 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:15.326210    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb"
Jul 29 23:12:18.055447 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:18.055387    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59" podUID="586ff4c9-ea54-4a54-8289-4add8b20d810" containerName="main" containerID="cri-o://eced9b9b5ba05d21ded30b15551f00ffe8068ffa6f37766863df428f175afe6d" gracePeriod=45
Jul 29 23:12:18.303872 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:18.303751    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59"
Jul 29 23:12:18.414969 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:18.414930    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmms8\" (UniqueName: \"kubernetes.io/projected/586ff4c9-ea54-4a54-8289-4add8b20d810-kube-api-access-qmms8\") pod \"586ff4c9-ea54-4a54-8289-4add8b20d810\" (UID: \"586ff4c9-ea54-4a54-8289-4add8b20d810\") "
Jul 29 23:12:18.415151 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:18.415057    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586ff4c9-ea54-4a54-8289-4add8b20d810-tls-certs\") pod \"586ff4c9-ea54-4a54-8289-4add8b20d810\" (UID: \"586ff4c9-ea54-4a54-8289-4add8b20d810\") "
Jul 29 23:12:18.417185 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:18.417147    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586ff4c9-ea54-4a54-8289-4add8b20d810-kube-api-access-qmms8" (OuterVolumeSpecName: "kube-api-access-qmms8") pod "586ff4c9-ea54-4a54-8289-4add8b20d810" (UID: "586ff4c9-ea54-4a54-8289-4add8b20d810"). InnerVolumeSpecName "kube-api-access-qmms8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:12:18.417185 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:18.417175    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/586ff4c9-ea54-4a54-8289-4add8b20d810-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "586ff4c9-ea54-4a54-8289-4add8b20d810" (UID: "586ff4c9-ea54-4a54-8289-4add8b20d810"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:12:18.473659 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:18.473625    2581 generic.go:356] "Generic (PLEG): container finished" podID="586ff4c9-ea54-4a54-8289-4add8b20d810" containerID="eced9b9b5ba05d21ded30b15551f00ffe8068ffa6f37766863df428f175afe6d" exitCode=0
Jul 29 23:12:18.473835 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:18.473673    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59"
Jul 29 23:12:18.473835 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:18.473714    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59" event={"ID":"586ff4c9-ea54-4a54-8289-4add8b20d810","Type":"ContainerDied","Data":"eced9b9b5ba05d21ded30b15551f00ffe8068ffa6f37766863df428f175afe6d"}
Jul 29 23:12:18.473835 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:18.473793    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59" event={"ID":"586ff4c9-ea54-4a54-8289-4add8b20d810","Type":"ContainerDied","Data":"d472f8f8c8c15ea028fed89a095f353735bc7c0ed1bca0ecc8f27fbe58504881"}
Jul 29 23:12:18.474022 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:18.473840    2581 scope.go:117] "RemoveContainer" containerID="eced9b9b5ba05d21ded30b15551f00ffe8068ffa6f37766863df428f175afe6d"
Jul 29 23:12:18.482637 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:18.482620    2581 scope.go:117] "RemoveContainer" containerID="eced9b9b5ba05d21ded30b15551f00ffe8068ffa6f37766863df428f175afe6d"
Jul 29 23:12:18.482940 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:12:18.482916    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eced9b9b5ba05d21ded30b15551f00ffe8068ffa6f37766863df428f175afe6d\": container with ID starting with eced9b9b5ba05d21ded30b15551f00ffe8068ffa6f37766863df428f175afe6d not found: ID does not exist" containerID="eced9b9b5ba05d21ded30b15551f00ffe8068ffa6f37766863df428f175afe6d"
Jul 29 23:12:18.483039 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:18.482952    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eced9b9b5ba05d21ded30b15551f00ffe8068ffa6f37766863df428f175afe6d"} err="failed to get container status \"eced9b9b5ba05d21ded30b15551f00ffe8068ffa6f37766863df428f175afe6d\": rpc error: code = NotFound desc = could not find container \"eced9b9b5ba05d21ded30b15551f00ffe8068ffa6f37766863df428f175afe6d\": container with ID starting with eced9b9b5ba05d21ded30b15551f00ffe8068ffa6f37766863df428f175afe6d not found: ID does not exist"
Jul 29 23:12:18.499665 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:18.499639    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59"]
Jul 29 23:12:18.502734 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:18.502712    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc7krt59"]
Jul 29 23:12:18.516465 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:18.516444    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586ff4c9-ea54-4a54-8289-4add8b20d810-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:12:18.516573 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:18.516471    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmms8\" (UniqueName: \"kubernetes.io/projected/586ff4c9-ea54-4a54-8289-4add8b20d810-kube-api-access-qmms8\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:12:19.952989 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:19.952956    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="586ff4c9-ea54-4a54-8289-4add8b20d810" path="/var/lib/kubelet/pods/586ff4c9-ea54-4a54-8289-4add8b20d810/volumes"
Jul 29 23:12:23.127483 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:23.127451    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:12:23.233340 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:23.233312    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:12:29.399383 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:29.399350    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:12:29.515415 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:29.515384    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:12:29.729663 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:29.729576    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:12:34.857715 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:34.857667    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:12:34.960570 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:34.960539    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:12:36.085675 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:12:36.085642    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:13:04.528416 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:13:04.528385    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:13:05.700793 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:13:05.700763    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:13:05.796071 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:13:05.796044    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:13:25.718694 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:13:25.718659    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:13:25.814154 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:13:25.814119    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:13:31.595557 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:13:31.595516    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:13:33.229611 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:13:33.229576    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:13:33.463902 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:13:33.463870    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:13:37.176535 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:13:37.176497    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:13:38.685310 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:13:38.685268    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:13:42.035149 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:13:42.035112    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:13:54.002665 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:13:54.002615    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:13:59.728034 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:13:59.727964    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb"]
Jul 29 23:13:59.733813 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:13:59.733770    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:13:59.801270 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:13:59.801237    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:05.326375 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:05.326336    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb" podUID="46e96b4d-271c-4448-954c-70ed1e9a70ff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:14:06.873793 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:06.873758    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:06.973400 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:06.973360    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:14.747724 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:14.747687    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb" podUID="46e96b4d-271c-4448-954c-70ed1e9a70ff" containerName="main" containerID="cri-o://0724688b48324f503ec95cfd523a89df514b7a2d4163cd430cc57dfae8ee65e5" gracePeriod=45
Jul 29 23:14:15.003502 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:15.003441    2581 generic.go:356] "Generic (PLEG): container finished" podID="46e96b4d-271c-4448-954c-70ed1e9a70ff" containerID="0724688b48324f503ec95cfd523a89df514b7a2d4163cd430cc57dfae8ee65e5" exitCode=0
Jul 29 23:14:15.003502 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:15.003490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb" event={"ID":"46e96b4d-271c-4448-954c-70ed1e9a70ff","Type":"ContainerDied","Data":"0724688b48324f503ec95cfd523a89df514b7a2d4163cd430cc57dfae8ee65e5"}
Jul 29 23:14:15.003642 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:15.003540    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb" event={"ID":"46e96b4d-271c-4448-954c-70ed1e9a70ff","Type":"ContainerDied","Data":"872ce6cf2aa793256578c0951764596837c32a8faed67e3fa68d1fd7fe3eada5"}
Jul 29 23:14:15.003642 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:15.003565    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="872ce6cf2aa793256578c0951764596837c32a8faed67e3fa68d1fd7fe3eada5"
Jul 29 23:14:15.007232 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:15.007213    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb"
Jul 29 23:14:15.088432 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:15.088404    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e96b4d-271c-4448-954c-70ed1e9a70ff-tls-certs\") pod \"46e96b4d-271c-4448-954c-70ed1e9a70ff\" (UID: \"46e96b4d-271c-4448-954c-70ed1e9a70ff\") "
Jul 29 23:14:15.088585 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:15.088471    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgvql\" (UniqueName: \"kubernetes.io/projected/46e96b4d-271c-4448-954c-70ed1e9a70ff-kube-api-access-lgvql\") pod \"46e96b4d-271c-4448-954c-70ed1e9a70ff\" (UID: \"46e96b4d-271c-4448-954c-70ed1e9a70ff\") "
Jul 29 23:14:15.090632 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:15.090593    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e96b4d-271c-4448-954c-70ed1e9a70ff-kube-api-access-lgvql" (OuterVolumeSpecName: "kube-api-access-lgvql") pod "46e96b4d-271c-4448-954c-70ed1e9a70ff" (UID: "46e96b4d-271c-4448-954c-70ed1e9a70ff"). InnerVolumeSpecName "kube-api-access-lgvql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:14:15.090741 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:15.090629    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46e96b4d-271c-4448-954c-70ed1e9a70ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "46e96b4d-271c-4448-954c-70ed1e9a70ff" (UID: "46e96b4d-271c-4448-954c-70ed1e9a70ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:14:15.189389 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:15.189361    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e96b4d-271c-4448-954c-70ed1e9a70ff-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:14:15.189389 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:15.189390    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgvql\" (UniqueName: \"kubernetes.io/projected/46e96b4d-271c-4448-954c-70ed1e9a70ff-kube-api-access-lgvql\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:14:16.008185 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:16.008153    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb"
Jul 29 23:14:16.026125 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:16.026096    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb"]
Jul 29 23:14:16.030165 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:16.030142    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d9b95c655-vq6lb"]
Jul 29 23:14:17.952635 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:17.952603    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46e96b4d-271c-4448-954c-70ed1e9a70ff" path="/var/lib/kubelet/pods/46e96b4d-271c-4448-954c-70ed1e9a70ff/volumes"
Jul 29 23:14:19.751003 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:19.750966    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:19.899085 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:19.899052    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:25.605709 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:25.605677    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:26.513525 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:26.513485    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:26.576617 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:26.576587    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:26.882536 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:26.882503    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:26.975308 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:26.975254    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:29.738838 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:29.738787    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:29.980834 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:29.980786    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm"]
Jul 29 23:14:29.981147 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:29.981135    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46e96b4d-271c-4448-954c-70ed1e9a70ff" containerName="main"
Jul 29 23:14:29.981191 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:29.981148    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46e96b4d-271c-4448-954c-70ed1e9a70ff" containerName="main"
Jul 29 23:14:29.981191 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:29.981169    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="586ff4c9-ea54-4a54-8289-4add8b20d810" containerName="main"
Jul 29 23:14:29.981191 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:29.981174    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="586ff4c9-ea54-4a54-8289-4add8b20d810" containerName="main"
Jul 29 23:14:29.981287 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:29.981230    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="46e96b4d-271c-4448-954c-70ed1e9a70ff" containerName="main"
Jul 29 23:14:29.981287 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:29.981238    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="586ff4c9-ea54-4a54-8289-4add8b20d810" containerName="main"
Jul 29 23:14:29.984109 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:29.984093    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm"
Jul 29 23:14:29.989270 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:29.989178    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-epp-sa-dockercfg-jbl2x\""
Jul 29 23:14:29.989270 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:29.989221    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-kserve-self-signed-certs\""
Jul 29 23:14:29.989270 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:29.989232    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"default-dockercfg-t27js\""
Jul 29 23:14:29.989270 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:29.989246    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"openshift-service-ca.crt\""
Jul 29 23:14:29.989646 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:29.989280    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"kube-root-ca.crt\""
Jul 29 23:14:30.001891 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:30.001869    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm"]
Jul 29 23:14:30.031039 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:30.031014    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:30.111405 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:30.111374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g67c5\" (UniqueName: \"kubernetes.io/projected/7563198d-ccdd-4367-a81f-e3d726ef88ed-kube-api-access-g67c5\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm\" (UID: \"7563198d-ccdd-4367-a81f-e3d726ef88ed\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm"
Jul 29 23:14:30.111551 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:30.111414    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7563198d-ccdd-4367-a81f-e3d726ef88ed-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm\" (UID: \"7563198d-ccdd-4367-a81f-e3d726ef88ed\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm"
Jul 29 23:14:30.211889 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:30.211857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g67c5\" (UniqueName: \"kubernetes.io/projected/7563198d-ccdd-4367-a81f-e3d726ef88ed-kube-api-access-g67c5\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm\" (UID: \"7563198d-ccdd-4367-a81f-e3d726ef88ed\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm"
Jul 29 23:14:30.212031 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:30.211901    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7563198d-ccdd-4367-a81f-e3d726ef88ed-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm\" (UID: \"7563198d-ccdd-4367-a81f-e3d726ef88ed\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm"
Jul 29 23:14:30.214307 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:30.214283    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7563198d-ccdd-4367-a81f-e3d726ef88ed-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm\" (UID: \"7563198d-ccdd-4367-a81f-e3d726ef88ed\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm"
Jul 29 23:14:30.219920 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:30.219899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g67c5\" (UniqueName: \"kubernetes.io/projected/7563198d-ccdd-4367-a81f-e3d726ef88ed-kube-api-access-g67c5\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm\" (UID: \"7563198d-ccdd-4367-a81f-e3d726ef88ed\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm"
Jul 29 23:14:30.294170 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:30.294104    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm"
Jul 29 23:14:30.421303 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:30.421240    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm"]
Jul 29 23:14:30.423513 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:14:30.423486    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7563198d_ccdd_4367_a81f_e3d726ef88ed.slice/crio-5ad45413fe54c9d32c512a135d5dc7cdcac82b1bd1aa28f30b367f4f3df64237 WatchSource:0}: Error finding container 5ad45413fe54c9d32c512a135d5dc7cdcac82b1bd1aa28f30b367f4f3df64237: Status 404 returned error can't find the container with id 5ad45413fe54c9d32c512a135d5dc7cdcac82b1bd1aa28f30b367f4f3df64237
Jul 29 23:14:31.076562 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:31.076527    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm" event={"ID":"7563198d-ccdd-4367-a81f-e3d726ef88ed","Type":"ContainerStarted","Data":"9905dce5fcd82702f6bc326b19dd89ddd7481df1417c7c5c1f467192e38f23b9"}
Jul 29 23:14:31.076978 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:31.076578    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm" event={"ID":"7563198d-ccdd-4367-a81f-e3d726ef88ed","Type":"ContainerStarted","Data":"5ad45413fe54c9d32c512a135d5dc7cdcac82b1bd1aa28f30b367f4f3df64237"}
Jul 29 23:14:31.076978 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:31.076599    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm"
Jul 29 23:14:31.094843 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:31.094783    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm" podStartSLOduration=2.094768577 podStartE2EDuration="2.094768577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:14:31.093037526 +0000 UTC m=+1693.651382343" watchObservedRunningTime="2026-07-29 23:14:31.094768577 +0000 UTC m=+1693.653113371"
Jul 29 23:14:31.543554 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:31.543518    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:31.602014 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:31.601991    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:33.187968 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:33.187906    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:38.699189 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:38.699155    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:38.794213 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:38.794176    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:56.763036 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:56.763006    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"]
Jul 29 23:14:56.766555 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:56.766534    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:56.770045 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:56.770026    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 29 23:14:56.771374 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:56.771342    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 29 23:14:56.771374 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:56.771362    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-wvckp\""
Jul 29 23:14:56.771374 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:56.771370    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 29 23:14:56.790878 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:56.790855    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"]
Jul 29 23:14:56.844166 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:56.844141    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:56.938458 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:56.938431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:56.938580 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:56.938481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:56.938580 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:56.938539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:56.938657 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:56.938591    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zf9h\" (UniqueName: \"kubernetes.io/projected/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-kube-api-access-5zf9h\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:56.938657 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:56.938614    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:56.938657 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:56.938631    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:57.039229 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:57.039149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:57.039229 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:57.039194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:57.039424 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:57.039250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zf9h\" (UniqueName: \"kubernetes.io/projected/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-kube-api-access-5zf9h\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:57.039424 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:57.039283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:57.039424 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:57.039308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:57.039424 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:57.039379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:57.039706 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:57.039677    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:57.039879 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:57.039858    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:57.039977 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:57.039891    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:57.041611 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:57.041588    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:57.041740 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:57.041722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:57.052187 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:57.052167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zf9h\" (UniqueName: \"kubernetes.io/projected/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-kube-api-access-5zf9h\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:57.070649 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:57.070629    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:14:57.077596 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:57.077577    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:14:57.205297 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:57.205265    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"]
Jul 29 23:14:57.208157 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:14:57.208132    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f7b9e1_cfcf_4dd6_b47a_a134c7fa4a85.slice/crio-4f4762fe5d2faaa1d8a987cfbd2c0c7633827604a61cdb4dae1bfd5fa0d62c5e WatchSource:0}: Error finding container 4f4762fe5d2faaa1d8a987cfbd2c0c7633827604a61cdb4dae1bfd5fa0d62c5e: Status 404 returned error can't find the container with id 4f4762fe5d2faaa1d8a987cfbd2c0c7633827604a61cdb4dae1bfd5fa0d62c5e
Jul 29 23:14:57.209842 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:57.209828    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 23:14:58.204908 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:58.204873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp" event={"ID":"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85","Type":"ContainerStarted","Data":"4f4762fe5d2faaa1d8a987cfbd2c0c7633827604a61cdb4dae1bfd5fa0d62c5e"}
Jul 29 23:14:59.211334 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:59.211293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp" event={"ID":"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85","Type":"ContainerStarted","Data":"b64ca8d6eae6628733b7da4f75c0c072f7b538c8c205671d4b1308564897ff38"}
Jul 29 23:14:59.230058 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:14:59.230008    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp" podStartSLOduration=1.6417092279999999 podStartE2EDuration="3.229992584s" totalImagesPullingTime="1.588283356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:14:59.228222153 +0000 UTC m=+1721.786566998" watchObservedRunningTime="2026-07-29 23:14:59.229992584 +0000 UTC m=+1721.788337400"
Jul 29 23:15:00.078092 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:15:00.078060    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:15:02.083537 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:15:02.083501    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-n67pm"
Jul 29 23:15:07.078257 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:15:07.078217    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:15:07.090498 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:15:07.090472    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:15:07.259456 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:15:07.259429    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:15:10.839772 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:15:10.839717    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:15:19.111448 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:15:19.111414    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:16:30.447869 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:30.447835    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"]
Jul 29 23:16:30.457016 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:16:30.456979    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:16:30.457016 ip-10-0-128-242 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:16:30.457016 ip-10-0-128-242 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 29 23:16:30.457016 ip-10-0-128-242 kubenswrapper[2581]:  > containerID="b64ca8d6eae6628733b7da4f75c0c072f7b538c8c205671d4b1308564897ff38" cmd=["/bin/sleep","15"]
Jul 29 23:16:30.457271 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:30.457037    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp" message="" err=<
Jul 29 23:16:30.457271 ip-10-0-128-242 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:16:30.457271 ip-10-0-128-242 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 29 23:16:30.457271 ip-10-0-128-242 kubenswrapper[2581]:  >
Jul 29 23:16:30.457271 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:16:30.457048    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:16:30.457271 ip-10-0-128-242 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:16:30.457271 ip-10-0-128-242 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 29 23:16:30.457271 ip-10-0-128-242 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp" podUID="84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85" containerName="main" containerID="cri-o://b64ca8d6eae6628733b7da4f75c0c072f7b538c8c205671d4b1308564897ff38"
Jul 29 23:16:30.457271 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:30.457091    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp" podUID="84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85" containerName="main" containerID="cri-o://b64ca8d6eae6628733b7da4f75c0c072f7b538c8c205671d4b1308564897ff38" gracePeriod=60
Jul 29 23:16:30.463687 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:30.463663    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:16:30.535117 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:30.535088    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:16:30.635694 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:30.635659    2581 generic.go:356] "Generic (PLEG): container finished" podID="84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85" containerID="b64ca8d6eae6628733b7da4f75c0c072f7b538c8c205671d4b1308564897ff38" exitCode=0
Jul 29 23:16:30.635897 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:30.635837    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp" event={"ID":"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85","Type":"ContainerDied","Data":"b64ca8d6eae6628733b7da4f75c0c072f7b538c8c205671d4b1308564897ff38"}
Jul 29 23:16:31.207348 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.207320    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:16:31.345404 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.345355    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-tmp-dir\") pod \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") "
Jul 29 23:16:31.345602 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.345412    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-model-cache\") pod \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") "
Jul 29 23:16:31.345602 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.345438    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-home\") pod \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") "
Jul 29 23:16:31.345602 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.345472    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zf9h\" (UniqueName: \"kubernetes.io/projected/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-kube-api-access-5zf9h\") pod \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") "
Jul 29 23:16:31.345602 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.345500    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-tls-certs\") pod \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") "
Jul 29 23:16:31.345602 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.345555    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-dshm\") pod \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\" (UID: \"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85\") "
Jul 29 23:16:31.345869 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.345681    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85" (UID: "84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:16:31.345869 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.345735    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-home" (OuterVolumeSpecName: "home") pod "84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85" (UID: "84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:16:31.345869 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.345821    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-model-cache" (OuterVolumeSpecName: "model-cache") pod "84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85" (UID: "84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:16:31.345986 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.345865    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-tmp-dir\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:16:31.345986 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.345888    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-home\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:16:31.347711 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.347689    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85" (UID: "84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:16:31.347775 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.347738    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-kube-api-access-5zf9h" (OuterVolumeSpecName: "kube-api-access-5zf9h") pod "84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85" (UID: "84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85"). InnerVolumeSpecName "kube-api-access-5zf9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:16:31.347841 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.347769    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-dshm" (OuterVolumeSpecName: "dshm") pod "84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85" (UID: "84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:16:31.447127 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.447094    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-model-cache\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:16:31.447127 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.447127    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zf9h\" (UniqueName: \"kubernetes.io/projected/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-kube-api-access-5zf9h\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:16:31.447330 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.447147    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:16:31.447330 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.447161    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85-dshm\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:16:31.644765 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.644686    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"
Jul 29 23:16:31.645218 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.644681    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp" event={"ID":"84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85","Type":"ContainerDied","Data":"4f4762fe5d2faaa1d8a987cfbd2c0c7633827604a61cdb4dae1bfd5fa0d62c5e"}
Jul 29 23:16:31.645218 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.644833    2581 scope.go:117] "RemoveContainer" containerID="b64ca8d6eae6628733b7da4f75c0c072f7b538c8c205671d4b1308564897ff38"
Jul 29 23:16:31.668016 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.667992    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"]
Jul 29 23:16:31.671135 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.671104    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-t9fzp"]
Jul 29 23:16:31.953354 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:31.953261    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85" path="/var/lib/kubelet/pods/84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85/volumes"
Jul 29 23:16:50.358433 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:50.358395    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:16:50.450592 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:50.450559    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:16:55.072075 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.072036    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"]
Jul 29 23:16:55.072496 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.072411    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85" containerName="main"
Jul 29 23:16:55.072496 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.072426    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85" containerName="main"
Jul 29 23:16:55.072615 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.072501    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="84f7b9e1-cfcf-4dd6-b47a-a134c7fa4a85" containerName="main"
Jul 29 23:16:55.074502 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.074484    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.078120 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.078096    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-bhfrc\""
Jul 29 23:16:55.078310 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.078298    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 29 23:16:55.078548 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.078527    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 29 23:16:55.079637 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.079621    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-h8x6p\""
Jul 29 23:16:55.079750 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.079695    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 29 23:16:55.089730 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.089705    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"]
Jul 29 23:16:55.100105 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.100080    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:16:55.143543 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.143510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.143543 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.143544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f69m\" (UniqueName: \"kubernetes.io/projected/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-kube-api-access-6f69m\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.143792 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.143576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-home\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.143792 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.143658    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.143792 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.143698    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.143792 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.143727    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.244475 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.244440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.244684 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.244479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f69m\" (UniqueName: \"kubernetes.io/projected/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-kube-api-access-6f69m\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.244684 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.244603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-home\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.244684 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.244643    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.244684 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.244670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.244945 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.244692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.244992 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.244939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.244992 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.244964    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-home\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.245166 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.245137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.246925 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.246898    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.247142 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.247123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.253845 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.253826    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f69m\" (UniqueName: \"kubernetes.io/projected/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-kube-api-access-6f69m\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.384967 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.384938    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:55.407705 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.407673    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:16:55.520486 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.520463    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"]
Jul 29 23:16:55.522572 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:16:55.522546    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd77f733_59bf_49c9_bbe1_6eec9f5eec43.slice/crio-fb9f9f579340511079c0651d76091872dc99dd1ce2b1499b6a8b96526e0d23be WatchSource:0}: Error finding container fb9f9f579340511079c0651d76091872dc99dd1ce2b1499b6a8b96526e0d23be: Status 404 returned error can't find the container with id fb9f9f579340511079c0651d76091872dc99dd1ce2b1499b6a8b96526e0d23be
Jul 29 23:16:55.761723 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:55.761690    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n" event={"ID":"fd77f733-59bf-49c9-bbe1-6eec9f5eec43","Type":"ContainerStarted","Data":"fb9f9f579340511079c0651d76091872dc99dd1ce2b1499b6a8b96526e0d23be"}
Jul 29 23:16:57.293147 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:57.293067    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:16:57.773248 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:57.773213    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n" event={"ID":"fd77f733-59bf-49c9-bbe1-6eec9f5eec43","Type":"ContainerStarted","Data":"504f9ccec32b0cd6dcaee2c8e56f6f6abe679090bc28cef315c829492fca9860"}
Jul 29 23:16:57.773428 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:57.773300    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:16:58.779890 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:58.779852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n" event={"ID":"fd77f733-59bf-49c9-bbe1-6eec9f5eec43","Type":"ContainerStarted","Data":"9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3"}
Jul 29 23:16:58.801134 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:58.801078    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n" podStartSLOduration=2.393204217 podStartE2EDuration="3.801064082s" totalImagesPullingTime="1.407859865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:16:58.799532305 +0000 UTC m=+1841.357877122" watchObservedRunningTime="2026-07-29 23:16:58.801064082 +0000 UTC m=+1841.359408898"
Jul 29 23:16:59.385553 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:16:59.385523    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:17:02.859848 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:02.859790    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:17:02.960602 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:02.960570    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:17:05.385143 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:05.385112    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:17:05.397687 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:05.397659    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:17:05.822421 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:05.822344    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:17:06.832441 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:06.832409    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:17:10.363838 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:10.363768    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:17:20.382607 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:20.382568    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:17:28.885001 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:28.884965    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"]
Jul 29 23:17:28.888813 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:28.888769    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:17:28.896649 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:17:28.896619    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:17:28.896649 ip-10-0-128-242 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:17:28.896649 ip-10-0-128-242 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 29 23:17:28.896649 ip-10-0-128-242 kubenswrapper[2581]:  > containerID="9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3" cmd=["/bin/sleep","15"]
Jul 29 23:17:28.896900 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:28.896667    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n" message="" err=<
Jul 29 23:17:28.896900 ip-10-0-128-242 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:17:28.896900 ip-10-0-128-242 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 29 23:17:28.896900 ip-10-0-128-242 kubenswrapper[2581]:  >
Jul 29 23:17:28.896900 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:17:28.896679    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:17:28.896900 ip-10-0-128-242 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:17:28.896900 ip-10-0-128-242 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 29 23:17:28.896900 ip-10-0-128-242 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n" podUID="fd77f733-59bf-49c9-bbe1-6eec9f5eec43" containerName="main" containerID="cri-o://9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3"
Jul 29 23:17:28.896900 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:28.896724    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n" podUID="fd77f733-59bf-49c9-bbe1-6eec9f5eec43" containerName="main" containerID="cri-o://9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3" gracePeriod=60
Jul 29 23:17:28.965285 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:28.965259    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:17:29.113347 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.113318    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n" podUID="fd77f733-59bf-49c9-bbe1-6eec9f5eec43" containerName="llm-d-routing-sidecar" containerID="cri-o://504f9ccec32b0cd6dcaee2c8e56f6f6abe679090bc28cef315c829492fca9860" gracePeriod=60
Jul 29 23:17:29.363300 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.363277    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:17:29.433992 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.433961    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-model-cache\") pod \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") "
Jul 29 23:17:29.434151 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.434016    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-dshm\") pod \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") "
Jul 29 23:17:29.434151 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.434046    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-tmp-dir\") pod \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") "
Jul 29 23:17:29.434151 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.434115    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f69m\" (UniqueName: \"kubernetes.io/projected/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-kube-api-access-6f69m\") pod \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") "
Jul 29 23:17:29.434151 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.434144    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-home\") pod \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") "
Jul 29 23:17:29.434358 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.434186    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-tls-certs\") pod \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\" (UID: \"fd77f733-59bf-49c9-bbe1-6eec9f5eec43\") "
Jul 29 23:17:29.434358 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.434217    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd77f733-59bf-49c9-bbe1-6eec9f5eec43" (UID: "fd77f733-59bf-49c9-bbe1-6eec9f5eec43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:17:29.434459 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.434385    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-home" (OuterVolumeSpecName: "home") pod "fd77f733-59bf-49c9-bbe1-6eec9f5eec43" (UID: "fd77f733-59bf-49c9-bbe1-6eec9f5eec43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:17:29.434459 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.434394    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd77f733-59bf-49c9-bbe1-6eec9f5eec43" (UID: "fd77f733-59bf-49c9-bbe1-6eec9f5eec43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:17:29.434556 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.434469    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-tmp-dir\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:17:29.434556 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.434487    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-home\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:17:29.434556 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.434502    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-model-cache\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:17:29.436300 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.436280    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-dshm" (OuterVolumeSpecName: "dshm") pod "fd77f733-59bf-49c9-bbe1-6eec9f5eec43" (UID: "fd77f733-59bf-49c9-bbe1-6eec9f5eec43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:17:29.436402 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.436306    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd77f733-59bf-49c9-bbe1-6eec9f5eec43" (UID: "fd77f733-59bf-49c9-bbe1-6eec9f5eec43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:17:29.436402 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.436386    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-kube-api-access-6f69m" (OuterVolumeSpecName: "kube-api-access-6f69m") pod "fd77f733-59bf-49c9-bbe1-6eec9f5eec43" (UID: "fd77f733-59bf-49c9-bbe1-6eec9f5eec43"). InnerVolumeSpecName "kube-api-access-6f69m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:17:29.535747 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.535680    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-dshm\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:17:29.535747 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.535704    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f69m\" (UniqueName: \"kubernetes.io/projected/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-kube-api-access-6f69m\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:17:29.535747 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.535715    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd77f733-59bf-49c9-bbe1-6eec9f5eec43-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:17:29.929907 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.929871    2581 generic.go:356] "Generic (PLEG): container finished" podID="fd77f733-59bf-49c9-bbe1-6eec9f5eec43" containerID="9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3" exitCode=0
Jul 29 23:17:29.929907 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.929902    2581 generic.go:356] "Generic (PLEG): container finished" podID="fd77f733-59bf-49c9-bbe1-6eec9f5eec43" containerID="504f9ccec32b0cd6dcaee2c8e56f6f6abe679090bc28cef315c829492fca9860" exitCode=0
Jul 29 23:17:29.930534 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.929929    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"
Jul 29 23:17:29.930534 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.929959    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n" event={"ID":"fd77f733-59bf-49c9-bbe1-6eec9f5eec43","Type":"ContainerDied","Data":"9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3"}
Jul 29 23:17:29.930534 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.930032    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n" event={"ID":"fd77f733-59bf-49c9-bbe1-6eec9f5eec43","Type":"ContainerDied","Data":"504f9ccec32b0cd6dcaee2c8e56f6f6abe679090bc28cef315c829492fca9860"}
Jul 29 23:17:29.930534 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.930061    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n" event={"ID":"fd77f733-59bf-49c9-bbe1-6eec9f5eec43","Type":"ContainerDied","Data":"fb9f9f579340511079c0651d76091872dc99dd1ce2b1499b6a8b96526e0d23be"}
Jul 29 23:17:29.930534 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.930093    2581 scope.go:117] "RemoveContainer" containerID="9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3"
Jul 29 23:17:29.939603 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.939578    2581 scope.go:117] "RemoveContainer" containerID="504f9ccec32b0cd6dcaee2c8e56f6f6abe679090bc28cef315c829492fca9860"
Jul 29 23:17:29.947133 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.947113    2581 scope.go:117] "RemoveContainer" containerID="9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3"
Jul 29 23:17:29.947385 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:17:29.947365    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3\": container with ID starting with 9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3 not found: ID does not exist" containerID="9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3"
Jul 29 23:17:29.947451 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.947393    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3"} err="failed to get container status \"9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3\": rpc error: code = NotFound desc = could not find container \"9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3\": container with ID starting with 9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3 not found: ID does not exist"
Jul 29 23:17:29.947451 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.947410    2581 scope.go:117] "RemoveContainer" containerID="504f9ccec32b0cd6dcaee2c8e56f6f6abe679090bc28cef315c829492fca9860"
Jul 29 23:17:29.947664 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:17:29.947647    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504f9ccec32b0cd6dcaee2c8e56f6f6abe679090bc28cef315c829492fca9860\": container with ID starting with 504f9ccec32b0cd6dcaee2c8e56f6f6abe679090bc28cef315c829492fca9860 not found: ID does not exist" containerID="504f9ccec32b0cd6dcaee2c8e56f6f6abe679090bc28cef315c829492fca9860"
Jul 29 23:17:29.947712 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.947670    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504f9ccec32b0cd6dcaee2c8e56f6f6abe679090bc28cef315c829492fca9860"} err="failed to get container status \"504f9ccec32b0cd6dcaee2c8e56f6f6abe679090bc28cef315c829492fca9860\": rpc error: code = NotFound desc = could not find container \"504f9ccec32b0cd6dcaee2c8e56f6f6abe679090bc28cef315c829492fca9860\": container with ID starting with 504f9ccec32b0cd6dcaee2c8e56f6f6abe679090bc28cef315c829492fca9860 not found: ID does not exist"
Jul 29 23:17:29.947712 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.947685    2581 scope.go:117] "RemoveContainer" containerID="9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3"
Jul 29 23:17:29.947939 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.947919    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3"} err="failed to get container status \"9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3\": rpc error: code = NotFound desc = could not find container \"9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3\": container with ID starting with 9dbc955d51b759f5e83062d12f9cce855f729d3d2644d02f088ef8bdf78798c3 not found: ID does not exist"
Jul 29 23:17:29.948004 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.947940    2581 scope.go:117] "RemoveContainer" containerID="504f9ccec32b0cd6dcaee2c8e56f6f6abe679090bc28cef315c829492fca9860"
Jul 29 23:17:29.948177 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.948159    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504f9ccec32b0cd6dcaee2c8e56f6f6abe679090bc28cef315c829492fca9860"} err="failed to get container status \"504f9ccec32b0cd6dcaee2c8e56f6f6abe679090bc28cef315c829492fca9860\": rpc error: code = NotFound desc = could not find container \"504f9ccec32b0cd6dcaee2c8e56f6f6abe679090bc28cef315c829492fca9860\": container with ID starting with 504f9ccec32b0cd6dcaee2c8e56f6f6abe679090bc28cef315c829492fca9860 not found: ID does not exist"
Jul 29 23:17:29.952484 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.952464    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"]
Jul 29 23:17:29.957213 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:29.957194    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-l4n8n"]
Jul 29 23:17:31.952794 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:31.952761    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd77f733-59bf-49c9-bbe1-6eec9f5eec43" path="/var/lib/kubelet/pods/fd77f733-59bf-49c9-bbe1-6eec9f5eec43/volumes"
Jul 29 23:17:44.838194 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:44.838160    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:17:48.894006 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:48.893969    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:17:48.988911 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:48.988877    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:17:55.203348 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:55.203303    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:17:57.417459 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:17:57.417425    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:18:01.165135 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:01.165103    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:18:01.265660 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:01.265623    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:18:04.639344 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:04.639307    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:18:04.892525 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:04.892433    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs"]
Jul 29 23:18:04.893008 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:04.892993    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd77f733-59bf-49c9-bbe1-6eec9f5eec43" containerName="main"
Jul 29 23:18:04.893054 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:04.893011    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd77f733-59bf-49c9-bbe1-6eec9f5eec43" containerName="main"
Jul 29 23:18:04.893054 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:04.893024    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd77f733-59bf-49c9-bbe1-6eec9f5eec43" containerName="llm-d-routing-sidecar"
Jul 29 23:18:04.893054 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:04.893034    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd77f733-59bf-49c9-bbe1-6eec9f5eec43" containerName="llm-d-routing-sidecar"
Jul 29 23:18:04.893151 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:04.893135    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd77f733-59bf-49c9-bbe1-6eec9f5eec43" containerName="main"
Jul 29 23:18:04.893151 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:04.893145    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd77f733-59bf-49c9-bbe1-6eec9f5eec43" containerName="llm-d-routing-sidecar"
Jul 29 23:18:04.896419 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:04.896400    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs"
Jul 29 23:18:04.900832 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:04.900794    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-d4m7z\""
Jul 29 23:18:04.901019 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:04.900812    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 29 23:18:04.901095 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:04.900846    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-rtb5v\""
Jul 29 23:18:04.901161 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:04.900853    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 29 23:18:04.901220 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:04.900855    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 29 23:18:04.902675 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:04.902650    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs"]
Jul 29 23:18:04.907962 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:04.907937    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:18:04.920307 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:04.920281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htbm6\" (UniqueName: \"kubernetes.io/projected/ebb1581d-a9b7-4f5f-a415-60ff8480dbb6-kube-api-access-htbm6\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs\" (UID: \"ebb1581d-a9b7-4f5f-a415-60ff8480dbb6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs"
Jul 29 23:18:04.920419 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:04.920317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebb1581d-a9b7-4f5f-a415-60ff8480dbb6-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs\" (UID: \"ebb1581d-a9b7-4f5f-a415-60ff8480dbb6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs"
Jul 29 23:18:05.009548 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:05.009524    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:18:05.021672 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:05.021649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebb1581d-a9b7-4f5f-a415-60ff8480dbb6-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs\" (UID: \"ebb1581d-a9b7-4f5f-a415-60ff8480dbb6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs"
Jul 29 23:18:05.021785 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:05.021735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htbm6\" (UniqueName: \"kubernetes.io/projected/ebb1581d-a9b7-4f5f-a415-60ff8480dbb6-kube-api-access-htbm6\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs\" (UID: \"ebb1581d-a9b7-4f5f-a415-60ff8480dbb6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs"
Jul 29 23:18:05.024030 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:05.024011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebb1581d-a9b7-4f5f-a415-60ff8480dbb6-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs\" (UID: \"ebb1581d-a9b7-4f5f-a415-60ff8480dbb6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs"
Jul 29 23:18:05.030537 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:05.030513    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htbm6\" (UniqueName: \"kubernetes.io/projected/ebb1581d-a9b7-4f5f-a415-60ff8480dbb6-kube-api-access-htbm6\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs\" (UID: \"ebb1581d-a9b7-4f5f-a415-60ff8480dbb6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs"
Jul 29 23:18:05.208076 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:05.207991    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs"
Jul 29 23:18:05.341148 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:05.341122    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs"]
Jul 29 23:18:05.343478 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:18:05.343454    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb1581d_a9b7_4f5f_a415_60ff8480dbb6.slice/crio-6192838c2f6c400ef83b3806e899681a1e6fd48b929244b77fa18b49ab097772 WatchSource:0}: Error finding container 6192838c2f6c400ef83b3806e899681a1e6fd48b929244b77fa18b49ab097772: Status 404 returned error can't find the container with id 6192838c2f6c400ef83b3806e899681a1e6fd48b929244b77fa18b49ab097772
Jul 29 23:18:06.104120 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:06.104089    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs" event={"ID":"ebb1581d-a9b7-4f5f-a415-60ff8480dbb6","Type":"ContainerStarted","Data":"de06873d998e9f1f2141b58acc54dd8d562213aa33383044f37a53fde8080619"}
Jul 29 23:18:06.104499 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:06.104136    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs" event={"ID":"ebb1581d-a9b7-4f5f-a415-60ff8480dbb6","Type":"ContainerStarted","Data":"6192838c2f6c400ef83b3806e899681a1e6fd48b929244b77fa18b49ab097772"}
Jul 29 23:18:06.104499 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:06.104215    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs"
Jul 29 23:18:06.139637 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:06.139582    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs" podStartSLOduration=2.139564546 podStartE2EDuration="2.139564546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:18:06.1382395 +0000 UTC m=+1908.696584315" watchObservedRunningTime="2026-07-29 23:18:06.139564546 +0000 UTC m=+1908.697909361"
Jul 29 23:18:16.038157 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:16.038122    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:18:18.097548 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:18.097518    2581 scope.go:117] "RemoveContainer" containerID="0724688b48324f503ec95cfd523a89df514b7a2d4163cd430cc57dfae8ee65e5"
Jul 29 23:18:37.112116 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:18:37.112087    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs"
Jul 29 23:19:21.811863 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:21.811750    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:19:21.829126 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:21.829092    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs"]
Jul 29 23:19:23.549999 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:23.549961    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:19:27.111210 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:27.111169    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs" podUID="ebb1581d-a9b7-4f5f-a415-60ff8480dbb6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:19:36.857991 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:36.857943    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs" podUID="ebb1581d-a9b7-4f5f-a415-60ff8480dbb6" containerName="main" containerID="cri-o://de06873d998e9f1f2141b58acc54dd8d562213aa33383044f37a53fde8080619" gracePeriod=45
Jul 29 23:19:37.120492 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:37.120434    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs"
Jul 29 23:19:37.292901 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:37.292869    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebb1581d-a9b7-4f5f-a415-60ff8480dbb6-tls-certs\") pod \"ebb1581d-a9b7-4f5f-a415-60ff8480dbb6\" (UID: \"ebb1581d-a9b7-4f5f-a415-60ff8480dbb6\") "
Jul 29 23:19:37.293063 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:37.292944    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htbm6\" (UniqueName: \"kubernetes.io/projected/ebb1581d-a9b7-4f5f-a415-60ff8480dbb6-kube-api-access-htbm6\") pod \"ebb1581d-a9b7-4f5f-a415-60ff8480dbb6\" (UID: \"ebb1581d-a9b7-4f5f-a415-60ff8480dbb6\") "
Jul 29 23:19:37.295179 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:37.295151    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb1581d-a9b7-4f5f-a415-60ff8480dbb6-kube-api-access-htbm6" (OuterVolumeSpecName: "kube-api-access-htbm6") pod "ebb1581d-a9b7-4f5f-a415-60ff8480dbb6" (UID: "ebb1581d-a9b7-4f5f-a415-60ff8480dbb6"). InnerVolumeSpecName "kube-api-access-htbm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:19:37.295281 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:37.295224    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebb1581d-a9b7-4f5f-a415-60ff8480dbb6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ebb1581d-a9b7-4f5f-a415-60ff8480dbb6" (UID: "ebb1581d-a9b7-4f5f-a415-60ff8480dbb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:19:37.393852 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:37.393778    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebb1581d-a9b7-4f5f-a415-60ff8480dbb6-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:19:37.393852 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:37.393820    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htbm6\" (UniqueName: \"kubernetes.io/projected/ebb1581d-a9b7-4f5f-a415-60ff8480dbb6-kube-api-access-htbm6\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:19:37.526950 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:37.526922    2581 generic.go:356] "Generic (PLEG): container finished" podID="ebb1581d-a9b7-4f5f-a415-60ff8480dbb6" containerID="de06873d998e9f1f2141b58acc54dd8d562213aa33383044f37a53fde8080619" exitCode=0
Jul 29 23:19:37.527105 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:37.526967    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs"
Jul 29 23:19:37.527105 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:37.527002    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs" event={"ID":"ebb1581d-a9b7-4f5f-a415-60ff8480dbb6","Type":"ContainerDied","Data":"de06873d998e9f1f2141b58acc54dd8d562213aa33383044f37a53fde8080619"}
Jul 29 23:19:37.527105 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:37.527055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs" event={"ID":"ebb1581d-a9b7-4f5f-a415-60ff8480dbb6","Type":"ContainerDied","Data":"6192838c2f6c400ef83b3806e899681a1e6fd48b929244b77fa18b49ab097772"}
Jul 29 23:19:37.527105 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:37.527082    2581 scope.go:117] "RemoveContainer" containerID="de06873d998e9f1f2141b58acc54dd8d562213aa33383044f37a53fde8080619"
Jul 29 23:19:37.538892 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:37.537965    2581 scope.go:117] "RemoveContainer" containerID="de06873d998e9f1f2141b58acc54dd8d562213aa33383044f37a53fde8080619"
Jul 29 23:19:37.538892 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:19:37.538275    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de06873d998e9f1f2141b58acc54dd8d562213aa33383044f37a53fde8080619\": container with ID starting with de06873d998e9f1f2141b58acc54dd8d562213aa33383044f37a53fde8080619 not found: ID does not exist" containerID="de06873d998e9f1f2141b58acc54dd8d562213aa33383044f37a53fde8080619"
Jul 29 23:19:37.538892 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:37.538305    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de06873d998e9f1f2141b58acc54dd8d562213aa33383044f37a53fde8080619"} err="failed to get container status \"de06873d998e9f1f2141b58acc54dd8d562213aa33383044f37a53fde8080619\": rpc error: code = NotFound desc = could not find container \"de06873d998e9f1f2141b58acc54dd8d562213aa33383044f37a53fde8080619\": container with ID starting with de06873d998e9f1f2141b58acc54dd8d562213aa33383044f37a53fde8080619 not found: ID does not exist"
Jul 29 23:19:37.556044 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:37.556022    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs"]
Jul 29 23:19:37.557489 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:37.557471    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs"]
Jul 29 23:19:37.952247 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:37.952217    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebb1581d-a9b7-4f5f-a415-60ff8480dbb6" path="/var/lib/kubelet/pods/ebb1581d-a9b7-4f5f-a415-60ff8480dbb6/volumes"
Jul 29 23:19:38.111110 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:38.111070    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5f8vxhs" podUID="ebb1581d-a9b7-4f5f-a415-60ff8480dbb6" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.46:9003\" within 1s: context deadline exceeded"
Jul 29 23:19:38.111270 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:19:38.111159    2581 logging.go:55] [core] [Channel #656 SubChannel #657]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.46:9003", ServerName: "10.132.0.46:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.46:9003: operation was canceled"
Jul 29 23:19:41.582997 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:41.582962    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:19:41.677471 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:41.677422    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:19:47.549253 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:47.549207    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:19:53.184253 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:53.184218    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:19:53.250434 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:53.250404    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:19:54.844037 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:54.843998    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:19:55.118528 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.118447    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24"]
Jul 29 23:19:55.119070 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.119054    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb1581d-a9b7-4f5f-a415-60ff8480dbb6" containerName="main"
Jul 29 23:19:55.119132 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.119074    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb1581d-a9b7-4f5f-a415-60ff8480dbb6" containerName="main"
Jul 29 23:19:55.119180 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.119169    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb1581d-a9b7-4f5f-a415-60ff8480dbb6" containerName="main"
Jul 29 23:19:55.122464 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.122444    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24"
Jul 29 23:19:55.126083 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.126058    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:19:55.126988 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.126967    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 29 23:19:55.127067 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.126969    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-9lq75\""
Jul 29 23:19:55.127067 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.127001    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 29 23:19:55.127067 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.127001    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-5hj4t\""
Jul 29 23:19:55.127205 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.126977    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 29 23:19:55.130722 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.130699    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24"]
Jul 29 23:19:55.217999 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.217963    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:19:55.236234 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.236210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12464ae-d5e9-4aae-bf14-9d13b03d7091-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24\" (UID: \"b12464ae-d5e9-4aae-bf14-9d13b03d7091\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24"
Jul 29 23:19:55.236355 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.236276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrxxq\" (UniqueName: \"kubernetes.io/projected/b12464ae-d5e9-4aae-bf14-9d13b03d7091-kube-api-access-jrxxq\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24\" (UID: \"b12464ae-d5e9-4aae-bf14-9d13b03d7091\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24"
Jul 29 23:19:55.337355 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.337317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrxxq\" (UniqueName: \"kubernetes.io/projected/b12464ae-d5e9-4aae-bf14-9d13b03d7091-kube-api-access-jrxxq\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24\" (UID: \"b12464ae-d5e9-4aae-bf14-9d13b03d7091\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24"
Jul 29 23:19:55.337517 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.337368    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12464ae-d5e9-4aae-bf14-9d13b03d7091-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24\" (UID: \"b12464ae-d5e9-4aae-bf14-9d13b03d7091\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24"
Jul 29 23:19:55.339774 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.339754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12464ae-d5e9-4aae-bf14-9d13b03d7091-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24\" (UID: \"b12464ae-d5e9-4aae-bf14-9d13b03d7091\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24"
Jul 29 23:19:55.345557 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.345531    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrxxq\" (UniqueName: \"kubernetes.io/projected/b12464ae-d5e9-4aae-bf14-9d13b03d7091-kube-api-access-jrxxq\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24\" (UID: \"b12464ae-d5e9-4aae-bf14-9d13b03d7091\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24"
Jul 29 23:19:55.433301 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.433212    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24"
Jul 29 23:19:55.564951 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.564905    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24"]
Jul 29 23:19:55.567373 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:19:55.567345    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12464ae_d5e9_4aae_bf14_9d13b03d7091.slice/crio-339d016de0d7329584f0e0f8253bdf7076de31d35eb267f698574bf8d3a2d69c WatchSource:0}: Error finding container 339d016de0d7329584f0e0f8253bdf7076de31d35eb267f698574bf8d3a2d69c: Status 404 returned error can't find the container with id 339d016de0d7329584f0e0f8253bdf7076de31d35eb267f698574bf8d3a2d69c
Jul 29 23:19:55.612638 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:55.612612    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24" event={"ID":"b12464ae-d5e9-4aae-bf14-9d13b03d7091","Type":"ContainerStarted","Data":"339d016de0d7329584f0e0f8253bdf7076de31d35eb267f698574bf8d3a2d69c"}
Jul 29 23:19:56.618851 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:56.618819    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24" event={"ID":"b12464ae-d5e9-4aae-bf14-9d13b03d7091","Type":"ContainerStarted","Data":"b2a90efc2b63b0681d9607eae5e46dec1afcca3230ba5c2980e47ae66be75c1b"}
Jul 29 23:19:56.619293 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:56.618916    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24"
Jul 29 23:19:56.635608 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:19:56.635561    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24" podStartSLOduration=1.635546137 podStartE2EDuration="1.635546137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:19:56.634708543 +0000 UTC m=+2019.193053365" watchObservedRunningTime="2026-07-29 23:19:56.635546137 +0000 UTC m=+2019.193890951"
Jul 29 23:20:10.127268 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:10.127224    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:20:17.398638 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:17.398606    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:20:23.386999 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:23.386960    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:20:27.625919 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:27.625888    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24"
Jul 29 23:20:33.959911 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:33.959870    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:20:33.965133 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:33.965104    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24"]
Jul 29 23:20:34.042062 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:34.042032    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:20:37.625108 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:37.625073    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24" podUID="b12464ae-d5e9-4aae-bf14-9d13b03d7091" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:20:47.625002 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:47.624960    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24" podUID="b12464ae-d5e9-4aae-bf14-9d13b03d7091" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:20:48.126962 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:48.126933    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:20:48.989063 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:48.989027    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24" podUID="b12464ae-d5e9-4aae-bf14-9d13b03d7091" containerName="main" containerID="cri-o://b2a90efc2b63b0681d9607eae5e46dec1afcca3230ba5c2980e47ae66be75c1b" gracePeriod=45
Jul 29 23:20:49.231009 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:49.230984    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24"
Jul 29 23:20:49.289103 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:49.289043    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12464ae-d5e9-4aae-bf14-9d13b03d7091-tls-certs\") pod \"b12464ae-d5e9-4aae-bf14-9d13b03d7091\" (UID: \"b12464ae-d5e9-4aae-bf14-9d13b03d7091\") "
Jul 29 23:20:49.289103 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:49.289093    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrxxq\" (UniqueName: \"kubernetes.io/projected/b12464ae-d5e9-4aae-bf14-9d13b03d7091-kube-api-access-jrxxq\") pod \"b12464ae-d5e9-4aae-bf14-9d13b03d7091\" (UID: \"b12464ae-d5e9-4aae-bf14-9d13b03d7091\") "
Jul 29 23:20:49.291090 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:49.291062    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b12464ae-d5e9-4aae-bf14-9d13b03d7091-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b12464ae-d5e9-4aae-bf14-9d13b03d7091" (UID: "b12464ae-d5e9-4aae-bf14-9d13b03d7091"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:20:49.291190 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:49.291122    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12464ae-d5e9-4aae-bf14-9d13b03d7091-kube-api-access-jrxxq" (OuterVolumeSpecName: "kube-api-access-jrxxq") pod "b12464ae-d5e9-4aae-bf14-9d13b03d7091" (UID: "b12464ae-d5e9-4aae-bf14-9d13b03d7091"). InnerVolumeSpecName "kube-api-access-jrxxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:20:49.389756 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:49.389719    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12464ae-d5e9-4aae-bf14-9d13b03d7091-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:20:49.389756 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:49.389751    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrxxq\" (UniqueName: \"kubernetes.io/projected/b12464ae-d5e9-4aae-bf14-9d13b03d7091-kube-api-access-jrxxq\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:20:49.865399 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:49.865369    2581 generic.go:356] "Generic (PLEG): container finished" podID="b12464ae-d5e9-4aae-bf14-9d13b03d7091" containerID="b2a90efc2b63b0681d9607eae5e46dec1afcca3230ba5c2980e47ae66be75c1b" exitCode=0
Jul 29 23:20:49.865608 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:49.865418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24" event={"ID":"b12464ae-d5e9-4aae-bf14-9d13b03d7091","Type":"ContainerDied","Data":"b2a90efc2b63b0681d9607eae5e46dec1afcca3230ba5c2980e47ae66be75c1b"}
Jul 29 23:20:49.865608 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:49.865423    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24"
Jul 29 23:20:49.865608 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:49.865459    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24" event={"ID":"b12464ae-d5e9-4aae-bf14-9d13b03d7091","Type":"ContainerDied","Data":"339d016de0d7329584f0e0f8253bdf7076de31d35eb267f698574bf8d3a2d69c"}
Jul 29 23:20:49.865608 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:49.865480    2581 scope.go:117] "RemoveContainer" containerID="b2a90efc2b63b0681d9607eae5e46dec1afcca3230ba5c2980e47ae66be75c1b"
Jul 29 23:20:49.875168 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:49.875151    2581 scope.go:117] "RemoveContainer" containerID="b2a90efc2b63b0681d9607eae5e46dec1afcca3230ba5c2980e47ae66be75c1b"
Jul 29 23:20:49.875434 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:20:49.875415    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a90efc2b63b0681d9607eae5e46dec1afcca3230ba5c2980e47ae66be75c1b\": container with ID starting with b2a90efc2b63b0681d9607eae5e46dec1afcca3230ba5c2980e47ae66be75c1b not found: ID does not exist" containerID="b2a90efc2b63b0681d9607eae5e46dec1afcca3230ba5c2980e47ae66be75c1b"
Jul 29 23:20:49.875485 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:49.875442    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a90efc2b63b0681d9607eae5e46dec1afcca3230ba5c2980e47ae66be75c1b"} err="failed to get container status \"b2a90efc2b63b0681d9607eae5e46dec1afcca3230ba5c2980e47ae66be75c1b\": rpc error: code = NotFound desc = could not find container \"b2a90efc2b63b0681d9607eae5e46dec1afcca3230ba5c2980e47ae66be75c1b\": container with ID starting with b2a90efc2b63b0681d9607eae5e46dec1afcca3230ba5c2980e47ae66be75c1b not found: ID does not exist"
Jul 29 23:20:49.886721 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:49.886697    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24"]
Jul 29 23:20:49.890592 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:49.890570    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bbczr24"]
Jul 29 23:20:49.952217 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:49.952195    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b12464ae-d5e9-4aae-bf14-9d13b03d7091" path="/var/lib/kubelet/pods/b12464ae-d5e9-4aae-bf14-9d13b03d7091/volumes"
Jul 29 23:20:53.974075 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:53.974046    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:20:54.073774 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:54.073737    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:20:59.448765 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.448722    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:20:59.630886 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.630855    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2"]
Jul 29 23:20:59.631269 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.631253    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b12464ae-d5e9-4aae-bf14-9d13b03d7091" containerName="main"
Jul 29 23:20:59.631354 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.631272    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b12464ae-d5e9-4aae-bf14-9d13b03d7091" containerName="main"
Jul 29 23:20:59.631414 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.631390    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b12464ae-d5e9-4aae-bf14-9d13b03d7091" containerName="main"
Jul 29 23:20:59.634294 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.634273    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2"
Jul 29 23:20:59.639147 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.639090    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 29 23:20:59.639256 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.639165    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-v2sjp\""
Jul 29 23:20:59.639256 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.639214    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 29 23:20:59.639368 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.639357    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-plplb\""
Jul 29 23:20:59.639458 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.639439    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 29 23:20:59.642137 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.642117    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2"]
Jul 29 23:20:59.654303 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.654283    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:20:59.666024 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.666002    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f91d65-d7e0-468a-9d1c-a96c2751c987-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2\" (UID: \"07f91d65-d7e0-468a-9d1c-a96c2751c987\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2"
Jul 29 23:20:59.666137 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.666039    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvh9s\" (UniqueName: \"kubernetes.io/projected/07f91d65-d7e0-468a-9d1c-a96c2751c987-kube-api-access-cvh9s\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2\" (UID: \"07f91d65-d7e0-468a-9d1c-a96c2751c987\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2"
Jul 29 23:20:59.766841 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.766760    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f91d65-d7e0-468a-9d1c-a96c2751c987-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2\" (UID: \"07f91d65-d7e0-468a-9d1c-a96c2751c987\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2"
Jul 29 23:20:59.766841 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.766824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvh9s\" (UniqueName: \"kubernetes.io/projected/07f91d65-d7e0-468a-9d1c-a96c2751c987-kube-api-access-cvh9s\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2\" (UID: \"07f91d65-d7e0-468a-9d1c-a96c2751c987\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2"
Jul 29 23:20:59.769135 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.769118    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f91d65-d7e0-468a-9d1c-a96c2751c987-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2\" (UID: \"07f91d65-d7e0-468a-9d1c-a96c2751c987\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2"
Jul 29 23:20:59.775442 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.775420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvh9s\" (UniqueName: \"kubernetes.io/projected/07f91d65-d7e0-468a-9d1c-a96c2751c987-kube-api-access-cvh9s\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2\" (UID: \"07f91d65-d7e0-468a-9d1c-a96c2751c987\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2"
Jul 29 23:20:59.945498 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:20:59.945475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2"
Jul 29 23:21:00.086058 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:00.085908    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2"]
Jul 29 23:21:00.088781 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:21:00.088753    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f91d65_d7e0_468a_9d1c_a96c2751c987.slice/crio-a6d04f946ded46938808059d0436c8d3a112d9b9e4cea23f6812ab4df2a55f07 WatchSource:0}: Error finding container a6d04f946ded46938808059d0436c8d3a112d9b9e4cea23f6812ab4df2a55f07: Status 404 returned error can't find the container with id a6d04f946ded46938808059d0436c8d3a112d9b9e4cea23f6812ab4df2a55f07
Jul 29 23:21:00.926054 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:00.926019    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2" event={"ID":"07f91d65-d7e0-468a-9d1c-a96c2751c987","Type":"ContainerStarted","Data":"03fb4e8f881e3479388ad98446cd9fefcb71306a9ccec786bce36ad06c507059"}
Jul 29 23:21:00.926432 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:00.926071    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2" event={"ID":"07f91d65-d7e0-468a-9d1c-a96c2751c987","Type":"ContainerStarted","Data":"a6d04f946ded46938808059d0436c8d3a112d9b9e4cea23f6812ab4df2a55f07"}
Jul 29 23:21:00.926432 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:00.926130    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2"
Jul 29 23:21:00.944947 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:00.944896    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2" podStartSLOduration=1.944883278 podStartE2EDuration="1.944883278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:21:00.943484404 +0000 UTC m=+2083.501829219" watchObservedRunningTime="2026-07-29 23:21:00.944883278 +0000 UTC m=+2083.503228093"
Jul 29 23:21:01.128118 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:01.128089    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:21:06.595045 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:06.595010    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:21:06.659495 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:06.659467    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:21:13.133154 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:13.133120    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:21:17.747975 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:17.747943    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:21:31.932721 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:31.932691    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2"
Jul 29 23:21:33.169316 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:33.168858    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:21:33.170900 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:33.170873    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2"]
Jul 29 23:21:33.264041 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:33.264007    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:21:41.932381 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:41.932342    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2" podUID="07f91d65-d7e0-468a-9d1c-a96c2751c987" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:21:48.192568 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:48.192512    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2" podUID="07f91d65-d7e0-468a-9d1c-a96c2751c987" containerName="main" containerID="cri-o://03fb4e8f881e3479388ad98446cd9fefcb71306a9ccec786bce36ad06c507059" gracePeriod=45
Jul 29 23:21:48.440717 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:48.440696    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2"
Jul 29 23:21:48.461640 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:48.461571    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f91d65-d7e0-468a-9d1c-a96c2751c987-tls-certs\") pod \"07f91d65-d7e0-468a-9d1c-a96c2751c987\" (UID: \"07f91d65-d7e0-468a-9d1c-a96c2751c987\") "
Jul 29 23:21:48.462022 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:48.462001    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvh9s\" (UniqueName: \"kubernetes.io/projected/07f91d65-d7e0-468a-9d1c-a96c2751c987-kube-api-access-cvh9s\") pod \"07f91d65-d7e0-468a-9d1c-a96c2751c987\" (UID: \"07f91d65-d7e0-468a-9d1c-a96c2751c987\") "
Jul 29 23:21:48.464061 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:48.464023    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f91d65-d7e0-468a-9d1c-a96c2751c987-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07f91d65-d7e0-468a-9d1c-a96c2751c987" (UID: "07f91d65-d7e0-468a-9d1c-a96c2751c987"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:21:48.464169 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:48.464133    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f91d65-d7e0-468a-9d1c-a96c2751c987-kube-api-access-cvh9s" (OuterVolumeSpecName: "kube-api-access-cvh9s") pod "07f91d65-d7e0-468a-9d1c-a96c2751c987" (UID: "07f91d65-d7e0-468a-9d1c-a96c2751c987"). InnerVolumeSpecName "kube-api-access-cvh9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:21:48.562692 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:48.562659    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvh9s\" (UniqueName: \"kubernetes.io/projected/07f91d65-d7e0-468a-9d1c-a96c2751c987-kube-api-access-cvh9s\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:21:48.562692 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:48.562691    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f91d65-d7e0-468a-9d1c-a96c2751c987-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:21:49.147424 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:49.147392    2581 generic.go:356] "Generic (PLEG): container finished" podID="07f91d65-d7e0-468a-9d1c-a96c2751c987" containerID="03fb4e8f881e3479388ad98446cd9fefcb71306a9ccec786bce36ad06c507059" exitCode=0
Jul 29 23:21:49.147592 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:49.147439    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2"
Jul 29 23:21:49.147592 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:49.147476    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2" event={"ID":"07f91d65-d7e0-468a-9d1c-a96c2751c987","Type":"ContainerDied","Data":"03fb4e8f881e3479388ad98446cd9fefcb71306a9ccec786bce36ad06c507059"}
Jul 29 23:21:49.147592 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:49.147536    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2" event={"ID":"07f91d65-d7e0-468a-9d1c-a96c2751c987","Type":"ContainerDied","Data":"a6d04f946ded46938808059d0436c8d3a112d9b9e4cea23f6812ab4df2a55f07"}
Jul 29 23:21:49.147592 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:49.147566    2581 scope.go:117] "RemoveContainer" containerID="03fb4e8f881e3479388ad98446cd9fefcb71306a9ccec786bce36ad06c507059"
Jul 29 23:21:49.156774 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:49.156754    2581 scope.go:117] "RemoveContainer" containerID="03fb4e8f881e3479388ad98446cd9fefcb71306a9ccec786bce36ad06c507059"
Jul 29 23:21:49.157062 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:21:49.157042    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03fb4e8f881e3479388ad98446cd9fefcb71306a9ccec786bce36ad06c507059\": container with ID starting with 03fb4e8f881e3479388ad98446cd9fefcb71306a9ccec786bce36ad06c507059 not found: ID does not exist" containerID="03fb4e8f881e3479388ad98446cd9fefcb71306a9ccec786bce36ad06c507059"
Jul 29 23:21:49.157115 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:49.157072    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03fb4e8f881e3479388ad98446cd9fefcb71306a9ccec786bce36ad06c507059"} err="failed to get container status \"03fb4e8f881e3479388ad98446cd9fefcb71306a9ccec786bce36ad06c507059\": rpc error: code = NotFound desc = could not find container \"03fb4e8f881e3479388ad98446cd9fefcb71306a9ccec786bce36ad06c507059\": container with ID starting with 03fb4e8f881e3479388ad98446cd9fefcb71306a9ccec786bce36ad06c507059 not found: ID does not exist"
Jul 29 23:21:49.169670 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:49.169646    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2"]
Jul 29 23:21:49.172963 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:49.172943    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebblh2"]
Jul 29 23:21:49.952551 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:49.952513    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07f91d65-d7e0-468a-9d1c-a96c2751c987" path="/var/lib/kubelet/pods/07f91d65-d7e0-468a-9d1c-a96c2751c987/volumes"
Jul 29 23:21:52.933083 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:52.933047    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:21:53.024794 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:53.024763    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:21:58.996942 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:58.996907    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:21:59.523046 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.523009    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:21:59.710933 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.710898    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5"]
Jul 29 23:21:59.711273 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.711260    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07f91d65-d7e0-468a-9d1c-a96c2751c987" containerName="main"
Jul 29 23:21:59.711320 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.711275    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="07f91d65-d7e0-468a-9d1c-a96c2751c987" containerName="main"
Jul 29 23:21:59.711367 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.711357    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="07f91d65-d7e0-468a-9d1c-a96c2751c987" containerName="main"
Jul 29 23:21:59.714446 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.714431    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5"
Jul 29 23:21:59.717043 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.717021    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 29 23:21:59.718532 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.718510    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-56mjh\""
Jul 29 23:21:59.718532 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.718515    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 29 23:21:59.718693 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.718550    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 29 23:21:59.718693 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.718588    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-6shz7\""
Jul 29 23:21:59.723489 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.723469    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5"]
Jul 29 23:21:59.739875 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.739854    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:21:59.745051 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.745023    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a356e49-dd4a-4405-ae7d-5115085ea65d-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5\" (UID: \"2a356e49-dd4a-4405-ae7d-5115085ea65d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5"
Jul 29 23:21:59.745146 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.745076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qvv9\" (UniqueName: \"kubernetes.io/projected/2a356e49-dd4a-4405-ae7d-5115085ea65d-kube-api-access-4qvv9\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5\" (UID: \"2a356e49-dd4a-4405-ae7d-5115085ea65d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5"
Jul 29 23:21:59.845481 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.845461    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a356e49-dd4a-4405-ae7d-5115085ea65d-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5\" (UID: \"2a356e49-dd4a-4405-ae7d-5115085ea65d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5"
Jul 29 23:21:59.845613 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.845503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qvv9\" (UniqueName: \"kubernetes.io/projected/2a356e49-dd4a-4405-ae7d-5115085ea65d-kube-api-access-4qvv9\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5\" (UID: \"2a356e49-dd4a-4405-ae7d-5115085ea65d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5"
Jul 29 23:21:59.847906 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.847888    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a356e49-dd4a-4405-ae7d-5115085ea65d-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5\" (UID: \"2a356e49-dd4a-4405-ae7d-5115085ea65d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5"
Jul 29 23:21:59.853580 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:21:59.853561    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qvv9\" (UniqueName: \"kubernetes.io/projected/2a356e49-dd4a-4405-ae7d-5115085ea65d-kube-api-access-4qvv9\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5\" (UID: \"2a356e49-dd4a-4405-ae7d-5115085ea65d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5"
Jul 29 23:22:00.025653 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:22:00.025619    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5"
Jul 29 23:22:00.162598 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:22:00.162569    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5"]
Jul 29 23:22:00.166681 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:22:00.166651    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a356e49_dd4a_4405_ae7d_5115085ea65d.slice/crio-b87d45bb0db0bb55915d2962552b65b20d1d949ccb95717b2a3126818c889067 WatchSource:0}: Error finding container b87d45bb0db0bb55915d2962552b65b20d1d949ccb95717b2a3126818c889067: Status 404 returned error can't find the container with id b87d45bb0db0bb55915d2962552b65b20d1d949ccb95717b2a3126818c889067
Jul 29 23:22:00.199967 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:22:00.199938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5" event={"ID":"2a356e49-dd4a-4405-ae7d-5115085ea65d","Type":"ContainerStarted","Data":"b87d45bb0db0bb55915d2962552b65b20d1d949ccb95717b2a3126818c889067"}
Jul 29 23:22:01.206586 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:22:01.206555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5" event={"ID":"2a356e49-dd4a-4405-ae7d-5115085ea65d","Type":"ContainerStarted","Data":"7463f5ba8b7dc1b1a41f43d2c46c6260a74f42ae323771cc038f8fc8d72d898d"}
Jul 29 23:22:01.206991 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:22:01.206666    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5"
Jul 29 23:22:01.223904 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:22:01.223851    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5" podStartSLOduration=2.223837315 podStartE2EDuration="2.223837315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:22:01.222633252 +0000 UTC m=+2143.780978069" watchObservedRunningTime="2026-07-29 23:22:01.223837315 +0000 UTC m=+2143.782182130"
Jul 29 23:22:04.629963 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:22:04.629926    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:22:04.693945 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:22:04.693914    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:22:16.280784 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:22:16.280707    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:22:32.213163 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:22:32.213128    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5"
Jul 29 23:22:41.420257 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:22:41.420224    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:22:41.617212 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:22:41.617184    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:22:48.800608 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:22:48.800577    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:22:48.854608 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:22:48.854579    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:22:50.687757 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:22:50.687719    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:22:56.288038 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:22:56.288002    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:23:15.107433 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:15.107393    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:23:15.164682 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:15.164649    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:23:35.053495 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:35.053461    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5"]
Jul 29 23:23:35.057084 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:35.057052    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:23:35.116710 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:35.116681    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:23:42.213299 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:42.213213    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5" podUID="2a356e49-dd4a-4405-ae7d-5115085ea65d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:23:50.072708 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:50.072651    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5" podUID="2a356e49-dd4a-4405-ae7d-5115085ea65d" containerName="main" containerID="cri-o://7463f5ba8b7dc1b1a41f43d2c46c6260a74f42ae323771cc038f8fc8d72d898d" gracePeriod=45
Jul 29 23:23:50.327701 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:50.327645    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5"
Jul 29 23:23:50.415478 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:50.415451    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qvv9\" (UniqueName: \"kubernetes.io/projected/2a356e49-dd4a-4405-ae7d-5115085ea65d-kube-api-access-4qvv9\") pod \"2a356e49-dd4a-4405-ae7d-5115085ea65d\" (UID: \"2a356e49-dd4a-4405-ae7d-5115085ea65d\") "
Jul 29 23:23:50.415630 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:50.415506    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a356e49-dd4a-4405-ae7d-5115085ea65d-tls-certs\") pod \"2a356e49-dd4a-4405-ae7d-5115085ea65d\" (UID: \"2a356e49-dd4a-4405-ae7d-5115085ea65d\") "
Jul 29 23:23:50.417740 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:50.417707    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a356e49-dd4a-4405-ae7d-5115085ea65d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a356e49-dd4a-4405-ae7d-5115085ea65d" (UID: "2a356e49-dd4a-4405-ae7d-5115085ea65d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:23:50.417740 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:50.417709    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a356e49-dd4a-4405-ae7d-5115085ea65d-kube-api-access-4qvv9" (OuterVolumeSpecName: "kube-api-access-4qvv9") pod "2a356e49-dd4a-4405-ae7d-5115085ea65d" (UID: "2a356e49-dd4a-4405-ae7d-5115085ea65d"). InnerVolumeSpecName "kube-api-access-4qvv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:23:50.516986 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:50.516962    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a356e49-dd4a-4405-ae7d-5115085ea65d-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:23:50.516986 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:50.516983    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qvv9\" (UniqueName: \"kubernetes.io/projected/2a356e49-dd4a-4405-ae7d-5115085ea65d-kube-api-access-4qvv9\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:23:50.714704 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:50.714675    2581 generic.go:356] "Generic (PLEG): container finished" podID="2a356e49-dd4a-4405-ae7d-5115085ea65d" containerID="7463f5ba8b7dc1b1a41f43d2c46c6260a74f42ae323771cc038f8fc8d72d898d" exitCode=0
Jul 29 23:23:50.714857 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:50.714722    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5"
Jul 29 23:23:50.714857 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:50.714761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5" event={"ID":"2a356e49-dd4a-4405-ae7d-5115085ea65d","Type":"ContainerDied","Data":"7463f5ba8b7dc1b1a41f43d2c46c6260a74f42ae323771cc038f8fc8d72d898d"}
Jul 29 23:23:50.714857 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:50.714834    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5" event={"ID":"2a356e49-dd4a-4405-ae7d-5115085ea65d","Type":"ContainerDied","Data":"b87d45bb0db0bb55915d2962552b65b20d1d949ccb95717b2a3126818c889067"}
Jul 29 23:23:50.714975 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:50.714860    2581 scope.go:117] "RemoveContainer" containerID="7463f5ba8b7dc1b1a41f43d2c46c6260a74f42ae323771cc038f8fc8d72d898d"
Jul 29 23:23:50.723494 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:50.723475    2581 scope.go:117] "RemoveContainer" containerID="7463f5ba8b7dc1b1a41f43d2c46c6260a74f42ae323771cc038f8fc8d72d898d"
Jul 29 23:23:50.723729 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:23:50.723709    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7463f5ba8b7dc1b1a41f43d2c46c6260a74f42ae323771cc038f8fc8d72d898d\": container with ID starting with 7463f5ba8b7dc1b1a41f43d2c46c6260a74f42ae323771cc038f8fc8d72d898d not found: ID does not exist" containerID="7463f5ba8b7dc1b1a41f43d2c46c6260a74f42ae323771cc038f8fc8d72d898d"
Jul 29 23:23:50.723792 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:50.723740    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7463f5ba8b7dc1b1a41f43d2c46c6260a74f42ae323771cc038f8fc8d72d898d"} err="failed to get container status \"7463f5ba8b7dc1b1a41f43d2c46c6260a74f42ae323771cc038f8fc8d72d898d\": rpc error: code = NotFound desc = could not find container \"7463f5ba8b7dc1b1a41f43d2c46c6260a74f42ae323771cc038f8fc8d72d898d\": container with ID starting with 7463f5ba8b7dc1b1a41f43d2c46c6260a74f42ae323771cc038f8fc8d72d898d not found: ID does not exist"
Jul 29 23:23:50.735613 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:50.735591    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5"]
Jul 29 23:23:50.739165 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:50.739146    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewzff5"]
Jul 29 23:23:51.952014 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:51.951986    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a356e49-dd4a-4405-ae7d-5115085ea65d" path="/var/lib/kubelet/pods/2a356e49-dd4a-4405-ae7d-5115085ea65d/volumes"
Jul 29 23:23:55.112004 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:55.111971    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:23:55.210477 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:23:55.210444    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:24:01.436724 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:01.436688    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:24:03.702312 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:03.702281    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:24:03.893229 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:03.893197    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq"]
Jul 29 23:24:03.893656 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:03.893640    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a356e49-dd4a-4405-ae7d-5115085ea65d" containerName="main"
Jul 29 23:24:03.893703 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:03.893659    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a356e49-dd4a-4405-ae7d-5115085ea65d" containerName="main"
Jul 29 23:24:03.893743 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:03.893731    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a356e49-dd4a-4405-ae7d-5115085ea65d" containerName="main"
Jul 29 23:24:03.896705 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:03.896690    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq"
Jul 29 23:24:03.901036 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:03.901010    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 29 23:24:03.901182 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:03.901029    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 29 23:24:03.901182 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:03.901079    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-x9ppl\""
Jul 29 23:24:03.901182 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:03.901099    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-dfbvj\""
Jul 29 23:24:03.901182 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:03.901068    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 29 23:24:03.905947 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:03.905927    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq"]
Jul 29 23:24:03.919268 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:03.919235    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:24:04.027504 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:04.027433    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkszc\" (UniqueName: \"kubernetes.io/projected/5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1-kube-api-access-gkszc\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq\" (UID: \"5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq"
Jul 29 23:24:04.027504 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:04.027475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq\" (UID: \"5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq"
Jul 29 23:24:04.128562 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:04.128538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq\" (UID: \"5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq"
Jul 29 23:24:04.128710 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:04.128653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkszc\" (UniqueName: \"kubernetes.io/projected/5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1-kube-api-access-gkszc\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq\" (UID: \"5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq"
Jul 29 23:24:04.130937 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:04.130911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq\" (UID: \"5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq"
Jul 29 23:24:04.137396 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:04.137370    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkszc\" (UniqueName: \"kubernetes.io/projected/5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1-kube-api-access-gkszc\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq\" (UID: \"5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq"
Jul 29 23:24:04.207916 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:04.207887    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq"
Jul 29 23:24:04.346066 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:04.346040    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq"]
Jul 29 23:24:04.348125 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:24:04.348094    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7536fd_2c0f_4d6e_88e8_834b9d4c8ec1.slice/crio-670791e1fb2f10075f385ba9e450675931d25803ef99edb05e82c70713a0f0d0 WatchSource:0}: Error finding container 670791e1fb2f10075f385ba9e450675931d25803ef99edb05e82c70713a0f0d0: Status 404 returned error can't find the container with id 670791e1fb2f10075f385ba9e450675931d25803ef99edb05e82c70713a0f0d0
Jul 29 23:24:04.787708 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:04.787669    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq" event={"ID":"5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1","Type":"ContainerStarted","Data":"13414491bceafb1cd536f595cd5556d59481a636e25110f09663bc221f70b69c"}
Jul 29 23:24:04.788374 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:04.788330    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq"
Jul 29 23:24:04.788506 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:04.788385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq" event={"ID":"5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1","Type":"ContainerStarted","Data":"670791e1fb2f10075f385ba9e450675931d25803ef99edb05e82c70713a0f0d0"}
Jul 29 23:24:04.806230 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:04.806184    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq" podStartSLOduration=1.806167815 podStartE2EDuration="1.806167815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:24:04.805669878 +0000 UTC m=+2267.364014693" watchObservedRunningTime="2026-07-29 23:24:04.806167815 +0000 UTC m=+2267.364512634"
Jul 29 23:24:06.927701 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:06.927623    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:24:06.996440 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:06.996407    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:24:16.113444 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:16.113414    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:24:16.170707 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:16.170666    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:24:17.533137 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:17.533090    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:24:35.794297 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:35.794264    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq"
Jul 29 23:24:48.327258 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:48.327201    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:24:48.522219 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:48.522187    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4"]
Jul 29 23:24:48.526233 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:48.526216    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4"
Jul 29 23:24:48.530312 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:48.530289    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 29 23:24:48.530453 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:48.530326    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-w899h\""
Jul 29 23:24:48.533088 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:48.533029    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4"]
Jul 29 23:24:48.681577 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:48.681543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcxkq\" (UniqueName: \"kubernetes.io/projected/d04e0902-0c76-454b-99c7-14215643154f-kube-api-access-kcxkq\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4\" (UID: \"d04e0902-0c76-454b-99c7-14215643154f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4"
Jul 29 23:24:48.681736 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:48.681598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d04e0902-0c76-454b-99c7-14215643154f-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4\" (UID: \"d04e0902-0c76-454b-99c7-14215643154f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4"
Jul 29 23:24:48.782486 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:48.782461    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d04e0902-0c76-454b-99c7-14215643154f-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4\" (UID: \"d04e0902-0c76-454b-99c7-14215643154f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4"
Jul 29 23:24:48.782625 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:48.782529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcxkq\" (UniqueName: \"kubernetes.io/projected/d04e0902-0c76-454b-99c7-14215643154f-kube-api-access-kcxkq\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4\" (UID: \"d04e0902-0c76-454b-99c7-14215643154f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4"
Jul 29 23:24:48.784956 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:48.784936    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d04e0902-0c76-454b-99c7-14215643154f-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4\" (UID: \"d04e0902-0c76-454b-99c7-14215643154f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4"
Jul 29 23:24:48.790071 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:48.790049    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcxkq\" (UniqueName: \"kubernetes.io/projected/d04e0902-0c76-454b-99c7-14215643154f-kube-api-access-kcxkq\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4\" (UID: \"d04e0902-0c76-454b-99c7-14215643154f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4"
Jul 29 23:24:48.838598 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:48.838574    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4"
Jul 29 23:24:48.965328 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:48.965256    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4"]
Jul 29 23:24:48.968778 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:24:48.968751    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04e0902_0c76_454b_99c7_14215643154f.slice/crio-ef49f398f3f156eda43cdb255bfb66a3323ec27c4fac4ac855db89764477661a WatchSource:0}: Error finding container ef49f398f3f156eda43cdb255bfb66a3323ec27c4fac4ac855db89764477661a: Status 404 returned error can't find the container with id ef49f398f3f156eda43cdb255bfb66a3323ec27c4fac4ac855db89764477661a
Jul 29 23:24:48.994622 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:48.994596    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4" event={"ID":"d04e0902-0c76-454b-99c7-14215643154f","Type":"ContainerStarted","Data":"ef49f398f3f156eda43cdb255bfb66a3323ec27c4fac4ac855db89764477661a"}
Jul 29 23:24:49.315756 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:49.315664    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:24:50.000349 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:50.000308    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4" event={"ID":"d04e0902-0c76-454b-99c7-14215643154f","Type":"ContainerStarted","Data":"14c4750f1c1068ff117e14377ef3158628b69d8aea6d63b07a959c293c9184fc"}
Jul 29 23:24:50.000767 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:50.000393    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4"
Jul 29 23:24:50.016534 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:24:50.016494    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4" podStartSLOduration=2.016480592 podStartE2EDuration="2.016480592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:24:50.01576272 +0000 UTC m=+2312.574107538" watchObservedRunningTime="2026-07-29 23:24:50.016480592 +0000 UTC m=+2312.574825410"
Jul 29 23:25:00.383982 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:00.383946    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:25:21.007175 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:21.007098    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4"
Jul 29 23:25:22.215229 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:22.215195    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4"]
Jul 29 23:25:22.216369 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:22.216334    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:25:22.278321 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:22.278289    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:25:31.007173 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:31.007139    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4" podUID="d04e0902-0c76-454b-99c7-14215643154f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:25:37.235312 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:37.235271    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4" podUID="d04e0902-0c76-454b-99c7-14215643154f" containerName="main" containerID="cri-o://14c4750f1c1068ff117e14377ef3158628b69d8aea6d63b07a959c293c9184fc" gracePeriod=45
Jul 29 23:25:37.488274 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:37.488211    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4"
Jul 29 23:25:37.579951 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:37.579925    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcxkq\" (UniqueName: \"kubernetes.io/projected/d04e0902-0c76-454b-99c7-14215643154f-kube-api-access-kcxkq\") pod \"d04e0902-0c76-454b-99c7-14215643154f\" (UID: \"d04e0902-0c76-454b-99c7-14215643154f\") "
Jul 29 23:25:37.579951 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:37.579953    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d04e0902-0c76-454b-99c7-14215643154f-tls-certs\") pod \"d04e0902-0c76-454b-99c7-14215643154f\" (UID: \"d04e0902-0c76-454b-99c7-14215643154f\") "
Jul 29 23:25:37.582033 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:37.582000    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04e0902-0c76-454b-99c7-14215643154f-kube-api-access-kcxkq" (OuterVolumeSpecName: "kube-api-access-kcxkq") pod "d04e0902-0c76-454b-99c7-14215643154f" (UID: "d04e0902-0c76-454b-99c7-14215643154f"). InnerVolumeSpecName "kube-api-access-kcxkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:25:37.582132 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:37.582057    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d04e0902-0c76-454b-99c7-14215643154f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d04e0902-0c76-454b-99c7-14215643154f" (UID: "d04e0902-0c76-454b-99c7-14215643154f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:25:37.680649 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:37.680613    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcxkq\" (UniqueName: \"kubernetes.io/projected/d04e0902-0c76-454b-99c7-14215643154f-kube-api-access-kcxkq\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:25:37.680649 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:37.680647    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d04e0902-0c76-454b-99c7-14215643154f-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:25:38.217242 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:38.217163    2581 generic.go:356] "Generic (PLEG): container finished" podID="d04e0902-0c76-454b-99c7-14215643154f" containerID="14c4750f1c1068ff117e14377ef3158628b69d8aea6d63b07a959c293c9184fc" exitCode=0
Jul 29 23:25:38.217242 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:38.217197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4" event={"ID":"d04e0902-0c76-454b-99c7-14215643154f","Type":"ContainerDied","Data":"14c4750f1c1068ff117e14377ef3158628b69d8aea6d63b07a959c293c9184fc"}
Jul 29 23:25:38.217242 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:38.217214    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4"
Jul 29 23:25:38.217462 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:38.217279    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4" event={"ID":"d04e0902-0c76-454b-99c7-14215643154f","Type":"ContainerDied","Data":"ef49f398f3f156eda43cdb255bfb66a3323ec27c4fac4ac855db89764477661a"}
Jul 29 23:25:38.217462 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:38.217305    2581 scope.go:117] "RemoveContainer" containerID="14c4750f1c1068ff117e14377ef3158628b69d8aea6d63b07a959c293c9184fc"
Jul 29 23:25:38.225698 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:38.225679    2581 scope.go:117] "RemoveContainer" containerID="14c4750f1c1068ff117e14377ef3158628b69d8aea6d63b07a959c293c9184fc"
Jul 29 23:25:38.225950 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:25:38.225929    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c4750f1c1068ff117e14377ef3158628b69d8aea6d63b07a959c293c9184fc\": container with ID starting with 14c4750f1c1068ff117e14377ef3158628b69d8aea6d63b07a959c293c9184fc not found: ID does not exist" containerID="14c4750f1c1068ff117e14377ef3158628b69d8aea6d63b07a959c293c9184fc"
Jul 29 23:25:38.226047 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:38.225955    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c4750f1c1068ff117e14377ef3158628b69d8aea6d63b07a959c293c9184fc"} err="failed to get container status \"14c4750f1c1068ff117e14377ef3158628b69d8aea6d63b07a959c293c9184fc\": rpc error: code = NotFound desc = could not find container \"14c4750f1c1068ff117e14377ef3158628b69d8aea6d63b07a959c293c9184fc\": container with ID starting with 14c4750f1c1068ff117e14377ef3158628b69d8aea6d63b07a959c293c9184fc not found: ID does not exist"
Jul 29 23:25:38.235745 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:38.235718    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4"]
Jul 29 23:25:38.237884 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:38.237864    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemqrc4"]
Jul 29 23:25:39.954953 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:39.954920    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d04e0902-0c76-454b-99c7-14215643154f" path="/var/lib/kubelet/pods/d04e0902-0c76-454b-99c7-14215643154f/volumes"
Jul 29 23:25:42.358678 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:42.358645    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq"]
Jul 29 23:25:42.363364 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:42.363331    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:25:42.428759 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:42.428730    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:25:45.793838 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:45.793785    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq" podUID="5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:25:55.794422 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:55.794383    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq" podUID="5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:25:57.378722 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:57.378683    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq" podUID="5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1" containerName="main" containerID="cri-o://13414491bceafb1cd536f595cd5556d59481a636e25110f09663bc221f70b69c" gracePeriod=45
Jul 29 23:25:57.640162 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:57.640107    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq"
Jul 29 23:25:57.740066 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:57.740035    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkszc\" (UniqueName: \"kubernetes.io/projected/5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1-kube-api-access-gkszc\") pod \"5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1\" (UID: \"5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1\") "
Jul 29 23:25:57.740234 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:57.740141    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1-tls-certs\") pod \"5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1\" (UID: \"5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1\") "
Jul 29 23:25:57.741954 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:57.741924    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1-kube-api-access-gkszc" (OuterVolumeSpecName: "kube-api-access-gkszc") pod "5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1" (UID: "5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1"). InnerVolumeSpecName "kube-api-access-gkszc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:25:57.742060 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:57.741977    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1" (UID: "5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:25:57.841215 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:57.841192    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:25:57.841215 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:57.841212    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkszc\" (UniqueName: \"kubernetes.io/projected/5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1-kube-api-access-gkszc\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:25:58.320540 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:58.320463    2581 generic.go:356] "Generic (PLEG): container finished" podID="5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1" containerID="13414491bceafb1cd536f595cd5556d59481a636e25110f09663bc221f70b69c" exitCode=0
Jul 29 23:25:58.320540 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:58.320508    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq"
Jul 29 23:25:58.320709 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:58.320546    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq" event={"ID":"5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1","Type":"ContainerDied","Data":"13414491bceafb1cd536f595cd5556d59481a636e25110f09663bc221f70b69c"}
Jul 29 23:25:58.320709 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:58.320601    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq" event={"ID":"5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1","Type":"ContainerDied","Data":"670791e1fb2f10075f385ba9e450675931d25803ef99edb05e82c70713a0f0d0"}
Jul 29 23:25:58.320709 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:58.320623    2581 scope.go:117] "RemoveContainer" containerID="13414491bceafb1cd536f595cd5556d59481a636e25110f09663bc221f70b69c"
Jul 29 23:25:58.329132 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:58.329114    2581 scope.go:117] "RemoveContainer" containerID="13414491bceafb1cd536f595cd5556d59481a636e25110f09663bc221f70b69c"
Jul 29 23:25:58.329371 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:25:58.329354    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13414491bceafb1cd536f595cd5556d59481a636e25110f09663bc221f70b69c\": container with ID starting with 13414491bceafb1cd536f595cd5556d59481a636e25110f09663bc221f70b69c not found: ID does not exist" containerID="13414491bceafb1cd536f595cd5556d59481a636e25110f09663bc221f70b69c"
Jul 29 23:25:58.329429 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:58.329379    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13414491bceafb1cd536f595cd5556d59481a636e25110f09663bc221f70b69c"} err="failed to get container status \"13414491bceafb1cd536f595cd5556d59481a636e25110f09663bc221f70b69c\": rpc error: code = NotFound desc = could not find container \"13414491bceafb1cd536f595cd5556d59481a636e25110f09663bc221f70b69c\": container with ID starting with 13414491bceafb1cd536f595cd5556d59481a636e25110f09663bc221f70b69c not found: ID does not exist"
Jul 29 23:25:58.339261 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:58.339242    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq"]
Jul 29 23:25:58.343121 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:58.343096    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7ttbq"]
Jul 29 23:25:59.952436 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:25:59.952407    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1" path="/var/lib/kubelet/pods/5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1/volumes"
Jul 29 23:26:02.431947 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:26:02.431909    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:26:02.525915 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:26:02.525884    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:26:08.757691 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:26:08.757662    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:26:09.139448 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:26:09.139421    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:26:09.371139 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:26:09.371105    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:26:14.196726 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:26:14.196687    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:26:14.258390 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:26:14.258361    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:26:29.993739 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:26:29.993702    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:27:37.572402 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:27:37.572363    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:28:04.365228 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:28:04.365190    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:28:04.421878 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:28:04.421842    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:28:24.348316 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:28:24.348237    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:28:24.442926 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:28:24.442879    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:28:28.207139 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:28:28.207103    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:28:28.471603 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:28:28.471514    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:28:30.817942 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:28:30.817899    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:28:36.460313 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:28:36.460281    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:28:36.560391 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:28:36.560361    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:28:49.497895 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:28:49.497851    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:28:49.784848 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:28:49.784744    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:28:57.388207 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:28:57.388166    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:28:57.440184 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:28:57.440156    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:30:00.175738 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:00.175705    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:30:01.942061 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:01.942027    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:30:20.186688 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:20.186654    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:30:20.343022 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:20.342993    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:30:20.443676 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:20.443605    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:30:20.543228 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:20.543199    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:30:22.821711 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:22.821673    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:30:22.873276 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:22.873241    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:30:23.493237 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:23.493206    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:30:26.299121 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:26.299087    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:30:31.804246 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:31.804194    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:30:31.866090 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:31.866050    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:30:52.731844 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:52.731792    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:30:52.956315 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:52.956284    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87"]
Jul 29 23:30:52.956821 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:52.956789    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1" containerName="main"
Jul 29 23:30:52.956876 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:52.956824    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1" containerName="main"
Jul 29 23:30:52.956876 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:52.956853    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d04e0902-0c76-454b-99c7-14215643154f" containerName="main"
Jul 29 23:30:52.956876 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:52.956862    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d04e0902-0c76-454b-99c7-14215643154f" containerName="main"
Jul 29 23:30:52.956981 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:52.956951    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d04e0902-0c76-454b-99c7-14215643154f" containerName="main"
Jul 29 23:30:52.956981 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:52.956967    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e7536fd-2c0f-4d6e-88e8-834b9d4c8ec1" containerName="main"
Jul 29 23:30:52.960443 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:52.960425    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87"
Jul 29 23:30:52.963185 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:52.963164    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 29 23:30:52.963302 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:52.963234    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 29 23:30:52.964554 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:52.964534    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-6fcvt\""
Jul 29 23:30:52.964554 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:52.964546    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 29 23:30:52.964726 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:52.964655    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-7txlv\""
Jul 29 23:30:52.969516 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:52.969494    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87"]
Jul 29 23:30:52.979925 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:52.979904    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:30:53.076308 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:53.076220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlthq\" (UniqueName: \"kubernetes.io/projected/060362de-281a-4a62-86ff-f3b3bd941dbb-kube-api-access-wlthq\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87\" (UID: \"060362de-281a-4a62-86ff-f3b3bd941dbb\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87"
Jul 29 23:30:53.076308 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:53.076266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060362de-281a-4a62-86ff-f3b3bd941dbb-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87\" (UID: \"060362de-281a-4a62-86ff-f3b3bd941dbb\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87"
Jul 29 23:30:53.177375 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:53.177335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlthq\" (UniqueName: \"kubernetes.io/projected/060362de-281a-4a62-86ff-f3b3bd941dbb-kube-api-access-wlthq\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87\" (UID: \"060362de-281a-4a62-86ff-f3b3bd941dbb\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87"
Jul 29 23:30:53.177375 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:53.177378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060362de-281a-4a62-86ff-f3b3bd941dbb-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87\" (UID: \"060362de-281a-4a62-86ff-f3b3bd941dbb\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87"
Jul 29 23:30:53.179858 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:53.179828    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060362de-281a-4a62-86ff-f3b3bd941dbb-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87\" (UID: \"060362de-281a-4a62-86ff-f3b3bd941dbb\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87"
Jul 29 23:30:53.185089 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:53.185071    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlthq\" (UniqueName: \"kubernetes.io/projected/060362de-281a-4a62-86ff-f3b3bd941dbb-kube-api-access-wlthq\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87\" (UID: \"060362de-281a-4a62-86ff-f3b3bd941dbb\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87"
Jul 29 23:30:53.271871 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:53.271847    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87"
Jul 29 23:30:53.404376 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:53.404348    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87"]
Jul 29 23:30:53.407638 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:30:53.407616    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060362de_281a_4a62_86ff_f3b3bd941dbb.slice/crio-9c31590cc257c8f04d4692b8b978982fc99947c83518d2ecf50cff0d6fe1bcb5 WatchSource:0}: Error finding container 9c31590cc257c8f04d4692b8b978982fc99947c83518d2ecf50cff0d6fe1bcb5: Status 404 returned error can't find the container with id 9c31590cc257c8f04d4692b8b978982fc99947c83518d2ecf50cff0d6fe1bcb5
Jul 29 23:30:53.681284 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:53.681181    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87" event={"ID":"060362de-281a-4a62-86ff-f3b3bd941dbb","Type":"ContainerStarted","Data":"dfbbba99584b9d6c351a92d87b26fcbf9c880482de082c706675a3e76847700b"}
Jul 29 23:30:53.681284 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:53.681239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87" event={"ID":"060362de-281a-4a62-86ff-f3b3bd941dbb","Type":"ContainerStarted","Data":"9c31590cc257c8f04d4692b8b978982fc99947c83518d2ecf50cff0d6fe1bcb5"}
Jul 29 23:30:53.681284 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:53.681270    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87"
Jul 29 23:30:53.702692 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:53.702623    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87" podStartSLOduration=1.7026013450000002 podStartE2EDuration="1.702601345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:30:53.700928662 +0000 UTC m=+2676.259273479" watchObservedRunningTime="2026-07-29 23:30:53.702601345 +0000 UTC m=+2676.260946162"
Jul 29 23:30:55.527866 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:30:55.527832    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:31:03.029554 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:31:03.029521    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:31:24.688083 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:31:24.687998    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87"
Jul 29 23:33:03.703995 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:03.703956    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87"]
Jul 29 23:33:03.730648 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:03.730615    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:33:03.819659 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:03.819626    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:33:04.687741 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:04.687705    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87" podUID="060362de-281a-4a62-86ff-f3b3bd941dbb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:33:14.687703 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:14.687662    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87" podUID="060362de-281a-4a62-86ff-f3b3bd941dbb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:33:18.730055 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:18.730011    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87" podUID="060362de-281a-4a62-86ff-f3b3bd941dbb" containerName="main" containerID="cri-o://dfbbba99584b9d6c351a92d87b26fcbf9c880482de082c706675a3e76847700b" gracePeriod=45
Jul 29 23:33:18.986577 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:18.986518    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87"
Jul 29 23:33:19.136539 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:19.136503    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlthq\" (UniqueName: \"kubernetes.io/projected/060362de-281a-4a62-86ff-f3b3bd941dbb-kube-api-access-wlthq\") pod \"060362de-281a-4a62-86ff-f3b3bd941dbb\" (UID: \"060362de-281a-4a62-86ff-f3b3bd941dbb\") "
Jul 29 23:33:19.136539 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:19.136551    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060362de-281a-4a62-86ff-f3b3bd941dbb-tls-certs\") pod \"060362de-281a-4a62-86ff-f3b3bd941dbb\" (UID: \"060362de-281a-4a62-86ff-f3b3bd941dbb\") "
Jul 29 23:33:19.138708 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:19.138680    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060362de-281a-4a62-86ff-f3b3bd941dbb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "060362de-281a-4a62-86ff-f3b3bd941dbb" (UID: "060362de-281a-4a62-86ff-f3b3bd941dbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:33:19.138708 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:19.138680    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060362de-281a-4a62-86ff-f3b3bd941dbb-kube-api-access-wlthq" (OuterVolumeSpecName: "kube-api-access-wlthq") pod "060362de-281a-4a62-86ff-f3b3bd941dbb" (UID: "060362de-281a-4a62-86ff-f3b3bd941dbb"). InnerVolumeSpecName "kube-api-access-wlthq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:33:19.237875 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:19.237819    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlthq\" (UniqueName: \"kubernetes.io/projected/060362de-281a-4a62-86ff-f3b3bd941dbb-kube-api-access-wlthq\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:33:19.237875 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:19.237840    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060362de-281a-4a62-86ff-f3b3bd941dbb-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:33:19.357319 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:19.357291    2581 generic.go:356] "Generic (PLEG): container finished" podID="060362de-281a-4a62-86ff-f3b3bd941dbb" containerID="dfbbba99584b9d6c351a92d87b26fcbf9c880482de082c706675a3e76847700b" exitCode=0
Jul 29 23:33:19.357427 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:19.357336    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87"
Jul 29 23:33:19.357427 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:19.357380    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87" event={"ID":"060362de-281a-4a62-86ff-f3b3bd941dbb","Type":"ContainerDied","Data":"dfbbba99584b9d6c351a92d87b26fcbf9c880482de082c706675a3e76847700b"}
Jul 29 23:33:19.357501 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:19.357429    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87" event={"ID":"060362de-281a-4a62-86ff-f3b3bd941dbb","Type":"ContainerDied","Data":"9c31590cc257c8f04d4692b8b978982fc99947c83518d2ecf50cff0d6fe1bcb5"}
Jul 29 23:33:19.357501 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:19.357455    2581 scope.go:117] "RemoveContainer" containerID="dfbbba99584b9d6c351a92d87b26fcbf9c880482de082c706675a3e76847700b"
Jul 29 23:33:19.365960 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:19.365938    2581 scope.go:117] "RemoveContainer" containerID="dfbbba99584b9d6c351a92d87b26fcbf9c880482de082c706675a3e76847700b"
Jul 29 23:33:19.366235 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:33:19.366213    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfbbba99584b9d6c351a92d87b26fcbf9c880482de082c706675a3e76847700b\": container with ID starting with dfbbba99584b9d6c351a92d87b26fcbf9c880482de082c706675a3e76847700b not found: ID does not exist" containerID="dfbbba99584b9d6c351a92d87b26fcbf9c880482de082c706675a3e76847700b"
Jul 29 23:33:19.366282 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:19.366245    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfbbba99584b9d6c351a92d87b26fcbf9c880482de082c706675a3e76847700b"} err="failed to get container status \"dfbbba99584b9d6c351a92d87b26fcbf9c880482de082c706675a3e76847700b\": rpc error: code = NotFound desc = could not find container \"dfbbba99584b9d6c351a92d87b26fcbf9c880482de082c706675a3e76847700b\": container with ID starting with dfbbba99584b9d6c351a92d87b26fcbf9c880482de082c706675a3e76847700b not found: ID does not exist"
Jul 29 23:33:19.379035 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:19.379013    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87"]
Jul 29 23:33:19.383026 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:19.383007    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schesdf87"]
Jul 29 23:33:19.952355 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:19.952321    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="060362de-281a-4a62-86ff-f3b3bd941dbb" path="/var/lib/kubelet/pods/060362de-281a-4a62-86ff-f3b3bd941dbb/volumes"
Jul 29 23:33:23.670463 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:23.670431    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:33:23.770395 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:23.770367    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:33:23.868870 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:23.868838    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:33:23.968780 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:23.968702    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:33:24.068512 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:24.068479    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:33:26.160258 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:26.160219    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:33:26.261224 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:26.261197    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:33:26.956096 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:26.956063    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:33:29.833094 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:29.833062    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:33:29.936466 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:29.936432    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:33:30.048593 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:30.048543    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:33:30.150676 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:30.150581    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:33:35.267062 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:35.267025    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:33:35.367897 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:35.367862    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:33:59.552841 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.552809    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"]
Jul 29 23:33:59.553320 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.553181    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="060362de-281a-4a62-86ff-f3b3bd941dbb" containerName="main"
Jul 29 23:33:59.553320 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.553198    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="060362de-281a-4a62-86ff-f3b3bd941dbb" containerName="main"
Jul 29 23:33:59.553422 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.553344    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="060362de-281a-4a62-86ff-f3b3bd941dbb" containerName="main"
Jul 29 23:33:59.556929 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.556909    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.559856 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.559826    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 29 23:33:59.559856 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.559835    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 29 23:33:59.560090 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.559890    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 29 23:33:59.560090 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.560085    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-sdqb6\""
Jul 29 23:33:59.571004 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.570976    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"]
Jul 29 23:33:59.581751 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.581730    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:33:59.658651 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.658627    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56094732-90fc-4ea7-b8cb-b551ca6cd72c-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" (UID: \"56094732-90fc-4ea7-b8cb-b551ca6cd72c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.658766 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.658672    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56094732-90fc-4ea7-b8cb-b551ca6cd72c-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" (UID: \"56094732-90fc-4ea7-b8cb-b551ca6cd72c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.658766 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.658692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56094732-90fc-4ea7-b8cb-b551ca6cd72c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" (UID: \"56094732-90fc-4ea7-b8cb-b551ca6cd72c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.658903 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.658823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n9r5\" (UniqueName: \"kubernetes.io/projected/56094732-90fc-4ea7-b8cb-b551ca6cd72c-kube-api-access-8n9r5\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" (UID: \"56094732-90fc-4ea7-b8cb-b551ca6cd72c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.658903 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.658857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56094732-90fc-4ea7-b8cb-b551ca6cd72c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" (UID: \"56094732-90fc-4ea7-b8cb-b551ca6cd72c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.658903 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.658875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56094732-90fc-4ea7-b8cb-b551ca6cd72c-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" (UID: \"56094732-90fc-4ea7-b8cb-b551ca6cd72c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.759915 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.759892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n9r5\" (UniqueName: \"kubernetes.io/projected/56094732-90fc-4ea7-b8cb-b551ca6cd72c-kube-api-access-8n9r5\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" (UID: \"56094732-90fc-4ea7-b8cb-b551ca6cd72c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.760058 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.759924    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56094732-90fc-4ea7-b8cb-b551ca6cd72c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" (UID: \"56094732-90fc-4ea7-b8cb-b551ca6cd72c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.760058 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.759941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56094732-90fc-4ea7-b8cb-b551ca6cd72c-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" (UID: \"56094732-90fc-4ea7-b8cb-b551ca6cd72c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.760058 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.759963    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56094732-90fc-4ea7-b8cb-b551ca6cd72c-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" (UID: \"56094732-90fc-4ea7-b8cb-b551ca6cd72c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.760058 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.759994    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56094732-90fc-4ea7-b8cb-b551ca6cd72c-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" (UID: \"56094732-90fc-4ea7-b8cb-b551ca6cd72c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.760058 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.760011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56094732-90fc-4ea7-b8cb-b551ca6cd72c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" (UID: \"56094732-90fc-4ea7-b8cb-b551ca6cd72c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.760336 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.760319    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56094732-90fc-4ea7-b8cb-b551ca6cd72c-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" (UID: \"56094732-90fc-4ea7-b8cb-b551ca6cd72c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.760398 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.760373    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56094732-90fc-4ea7-b8cb-b551ca6cd72c-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" (UID: \"56094732-90fc-4ea7-b8cb-b551ca6cd72c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.760441 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.760420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56094732-90fc-4ea7-b8cb-b551ca6cd72c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" (UID: \"56094732-90fc-4ea7-b8cb-b551ca6cd72c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.762144 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.762119    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56094732-90fc-4ea7-b8cb-b551ca6cd72c-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" (UID: \"56094732-90fc-4ea7-b8cb-b551ca6cd72c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.762363 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.762346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56094732-90fc-4ea7-b8cb-b551ca6cd72c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" (UID: \"56094732-90fc-4ea7-b8cb-b551ca6cd72c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.772881 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.772858    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n9r5\" (UniqueName: \"kubernetes.io/projected/56094732-90fc-4ea7-b8cb-b551ca6cd72c-kube-api-access-8n9r5\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" (UID: \"56094732-90fc-4ea7-b8cb-b551ca6cd72c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:33:59.790349 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:33:59.790323    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:34:12.492972 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:34:12.492885    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:34:14.501668 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:34:14.501633    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:34:14.648211 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:34:14.648174    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:34:40.682647 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:34:40.682613    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:34:40.783851 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:34:40.783815    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:36:02.571248 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:36:02.571208    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 29 23:36:03.125214 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:36:03.125184    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:36:56.960678 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:36:56.960645    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:38:06.126471 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:38:06.126436    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 29 23:38:06.696136 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:38:06.696099    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:38:42.102592 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:38:42.102486    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:40:09.697346 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:40:09.697258    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 29 23:40:10.247586 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:40:10.247554    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:40:18.902205 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:40:18.902167    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:42:13.248565 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:42:13.248524    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 29 23:42:13.809302 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:42:13.809267    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:44:16.809913 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:44:16.809871    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 29 23:44:17.375145 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:44:17.375112    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:44:54.159379 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:44:54.159326    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:46:20.376480 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:46:20.376445    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 29 23:46:20.939769 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:46:20.939736    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:48:23.940928 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:48:23.940881    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 29 23:48:24.501751 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:48:24.501722    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:48:46.190964 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:48:46.190917    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:48:46.292398 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:48:46.292332    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:48:46.389940 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:48:46.389904    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:48:46.491421 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:48:46.491330    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:48:48.612430 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:48:48.612394    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:48:48.713933 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:48:48.713882    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:48:49.428751 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:48:49.428718    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:49:18.329912 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:18.329837    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:49:18.428942 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:18.428898    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:49:18.540826 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:18.540773    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:49:18.694518 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:18.694480    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2"]
Jul 29 23:49:18.698157 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:18.698137    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2"
Jul 29 23:49:18.703216 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:18.703195    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 29 23:49:18.703329 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:18.703215    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 29 23:49:18.703329 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:18.703221    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-49rd4\""
Jul 29 23:49:18.703329 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:18.703267    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-vxspr\""
Jul 29 23:49:18.703329 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:18.703286    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 29 23:49:18.707716 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:18.707694    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2"]
Jul 29 23:49:18.711755 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:18.711722    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:49:18.821955 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:18.821933    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jxjl\" (UniqueName: \"kubernetes.io/projected/0a51c264-3fe9-41fc-a684-93ae06c45a1c-kube-api-access-9jxjl\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2\" (UID: \"0a51c264-3fe9-41fc-a684-93ae06c45a1c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2"
Jul 29 23:49:18.822082 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:18.821981    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a51c264-3fe9-41fc-a684-93ae06c45a1c-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2\" (UID: \"0a51c264-3fe9-41fc-a684-93ae06c45a1c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2"
Jul 29 23:49:18.923100 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:18.923070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a51c264-3fe9-41fc-a684-93ae06c45a1c-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2\" (UID: \"0a51c264-3fe9-41fc-a684-93ae06c45a1c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2"
Jul 29 23:49:18.923245 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:18.923164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jxjl\" (UniqueName: \"kubernetes.io/projected/0a51c264-3fe9-41fc-a684-93ae06c45a1c-kube-api-access-9jxjl\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2\" (UID: \"0a51c264-3fe9-41fc-a684-93ae06c45a1c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2"
Jul 29 23:49:18.925491 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:18.925468    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a51c264-3fe9-41fc-a684-93ae06c45a1c-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2\" (UID: \"0a51c264-3fe9-41fc-a684-93ae06c45a1c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2"
Jul 29 23:49:18.931074 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:18.931054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jxjl\" (UniqueName: \"kubernetes.io/projected/0a51c264-3fe9-41fc-a684-93ae06c45a1c-kube-api-access-9jxjl\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2\" (UID: \"0a51c264-3fe9-41fc-a684-93ae06c45a1c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2"
Jul 29 23:49:19.009072 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:19.008996    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2"
Jul 29 23:49:19.136981 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:19.136954    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2"]
Jul 29 23:49:19.138932 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:49:19.138900    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a51c264_3fe9_41fc_a684_93ae06c45a1c.slice/crio-2ec2072c1b9092f399fc7d5f6f0cf37dc44684edbc5136ea126319455a49aaa2 WatchSource:0}: Error finding container 2ec2072c1b9092f399fc7d5f6f0cf37dc44684edbc5136ea126319455a49aaa2: Status 404 returned error can't find the container with id 2ec2072c1b9092f399fc7d5f6f0cf37dc44684edbc5136ea126319455a49aaa2
Jul 29 23:49:19.757463 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:19.757427    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2" event={"ID":"0a51c264-3fe9-41fc-a684-93ae06c45a1c","Type":"ContainerStarted","Data":"e795c5cfc86fd397b90d6359a746e6a4b034dadc677caadbee4390307f9f0bff"}
Jul 29 23:49:19.757873 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:19.757494    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2"
Jul 29 23:49:19.757873 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:19.757520    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2" event={"ID":"0a51c264-3fe9-41fc-a684-93ae06c45a1c","Type":"ContainerStarted","Data":"2ec2072c1b9092f399fc7d5f6f0cf37dc44684edbc5136ea126319455a49aaa2"}
Jul 29 23:49:19.777927 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:19.777879    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2" podStartSLOduration=1.77786564 podStartE2EDuration="1.77786564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:49:19.776041117 +0000 UTC m=+3782.334385934" watchObservedRunningTime="2026-07-29 23:49:19.77786564 +0000 UTC m=+3782.336210456"
Jul 29 23:49:44.254519 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:44.254474    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:49:50.763400 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:49:50.763368    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2"
Jul 29 23:50:27.503075 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:50:27.503033    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 29 23:50:28.064904 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:50:28.064874    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:50:51.862851 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:50:51.862781    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:50:53.947261 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:50:53.947173    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:15.059430 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:15.059397    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:15.062870 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:15.062844    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2"]
Jul 29 23:51:15.155288 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:15.155241    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:20.763873 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:20.763831    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2" podUID="0a51c264-3fe9-41fc-a684-93ae06c45a1c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:51:30.082633 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:30.082588    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2" podUID="0a51c264-3fe9-41fc-a684-93ae06c45a1c" containerName="main" containerID="cri-o://e795c5cfc86fd397b90d6359a746e6a4b034dadc677caadbee4390307f9f0bff" gracePeriod=45
Jul 29 23:51:30.326297 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:30.326274    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2"
Jul 29 23:51:30.344931 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:30.344861    2581 generic.go:356] "Generic (PLEG): container finished" podID="0a51c264-3fe9-41fc-a684-93ae06c45a1c" containerID="e795c5cfc86fd397b90d6359a746e6a4b034dadc677caadbee4390307f9f0bff" exitCode=0
Jul 29 23:51:30.344931 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:30.344908    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2"
Jul 29 23:51:30.344931 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:30.344916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2" event={"ID":"0a51c264-3fe9-41fc-a684-93ae06c45a1c","Type":"ContainerDied","Data":"e795c5cfc86fd397b90d6359a746e6a4b034dadc677caadbee4390307f9f0bff"}
Jul 29 23:51:30.345154 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:30.344966    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2" event={"ID":"0a51c264-3fe9-41fc-a684-93ae06c45a1c","Type":"ContainerDied","Data":"2ec2072c1b9092f399fc7d5f6f0cf37dc44684edbc5136ea126319455a49aaa2"}
Jul 29 23:51:30.345154 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:30.344999    2581 scope.go:117] "RemoveContainer" containerID="e795c5cfc86fd397b90d6359a746e6a4b034dadc677caadbee4390307f9f0bff"
Jul 29 23:51:30.356684 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:30.356653    2581 scope.go:117] "RemoveContainer" containerID="e795c5cfc86fd397b90d6359a746e6a4b034dadc677caadbee4390307f9f0bff"
Jul 29 23:51:30.356960 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:51:30.356941    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e795c5cfc86fd397b90d6359a746e6a4b034dadc677caadbee4390307f9f0bff\": container with ID starting with e795c5cfc86fd397b90d6359a746e6a4b034dadc677caadbee4390307f9f0bff not found: ID does not exist" containerID="e795c5cfc86fd397b90d6359a746e6a4b034dadc677caadbee4390307f9f0bff"
Jul 29 23:51:30.357049 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:30.356967    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e795c5cfc86fd397b90d6359a746e6a4b034dadc677caadbee4390307f9f0bff"} err="failed to get container status \"e795c5cfc86fd397b90d6359a746e6a4b034dadc677caadbee4390307f9f0bff\": rpc error: code = NotFound desc = could not find container \"e795c5cfc86fd397b90d6359a746e6a4b034dadc677caadbee4390307f9f0bff\": container with ID starting with e795c5cfc86fd397b90d6359a746e6a4b034dadc677caadbee4390307f9f0bff not found: ID does not exist"
Jul 29 23:51:30.506224 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:30.506200    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jxjl\" (UniqueName: \"kubernetes.io/projected/0a51c264-3fe9-41fc-a684-93ae06c45a1c-kube-api-access-9jxjl\") pod \"0a51c264-3fe9-41fc-a684-93ae06c45a1c\" (UID: \"0a51c264-3fe9-41fc-a684-93ae06c45a1c\") "
Jul 29 23:51:30.506345 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:30.506265    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a51c264-3fe9-41fc-a684-93ae06c45a1c-tls-certs\") pod \"0a51c264-3fe9-41fc-a684-93ae06c45a1c\" (UID: \"0a51c264-3fe9-41fc-a684-93ae06c45a1c\") "
Jul 29 23:51:30.508369 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:30.508347    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a51c264-3fe9-41fc-a684-93ae06c45a1c-kube-api-access-9jxjl" (OuterVolumeSpecName: "kube-api-access-9jxjl") pod "0a51c264-3fe9-41fc-a684-93ae06c45a1c" (UID: "0a51c264-3fe9-41fc-a684-93ae06c45a1c"). InnerVolumeSpecName "kube-api-access-9jxjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:51:30.508482 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:30.508349    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a51c264-3fe9-41fc-a684-93ae06c45a1c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a51c264-3fe9-41fc-a684-93ae06c45a1c" (UID: "0a51c264-3fe9-41fc-a684-93ae06c45a1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:51:30.607197 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:30.607129    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a51c264-3fe9-41fc-a684-93ae06c45a1c-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:51:30.607197 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:30.607151    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jxjl\" (UniqueName: \"kubernetes.io/projected/0a51c264-3fe9-41fc-a684-93ae06c45a1c-kube-api-access-9jxjl\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:51:30.666353 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:30.666330    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2"]
Jul 29 23:51:30.669747 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:30.669727    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche5zxr2"]
Jul 29 23:51:31.952135 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:31.952106    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a51c264-3fe9-41fc-a684-93ae06c45a1c" path="/var/lib/kubelet/pods/0a51c264-3fe9-41fc-a684-93ae06c45a1c/volumes"
Jul 29 23:51:34.936324 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:34.936289    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:35.035636 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:35.035610    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:35.139008 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:35.138960    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:37.310500 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:37.310463    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:37.411374 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:37.411339    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:37.511266 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:37.511221    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:40.808118 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:40.808076    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:40.860534 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:40.860498    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:40.954029 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:40.954006    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:41.053820 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:41.053775    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:41.154548 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:41.154524    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:43.292055 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:43.292021    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:43.390893 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:43.390862    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:44.114857 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:44.114829    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:44.214970 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:44.214943    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:44.315808 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:44.315766    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:46.644554 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:46.644523    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:46.936120 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:46.936026    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:48.491246 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:48.491215    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:48.581519 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:48.581489    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:49.881033 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:49.880998    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:50.028873 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:50.028822    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:51.754047 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:51.754015    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:51.854616 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:51.854589    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:53.705338 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:53.705306    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:53.954723 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:53.954686    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:54.049629 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:54.049552    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:54.482477 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:54.482446    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:54.585036 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:54.585004    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:58.847456 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:58.847423    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:58.939686 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:58.939660    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:59.865733 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:59.865702    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:51:59.965639 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:51:59.965610    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:00.068478 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:00.068451    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:00.171013 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:00.170927    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:00.607377 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:00.607350    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:00.708471 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:00.708443    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:00.812723 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:00.812698    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:01.689158 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:01.689127    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:01.790027 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:01.789994    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:01.848488 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:01.848459    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:02.391812 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:02.391765    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:02.493129 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:02.493098    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:02.661706 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:02.661628    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:03.100199 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:03.100164    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:03.200206 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:03.200178    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:04.724422 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:04.724388    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:04.825314 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:04.825285    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:05.974580 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:05.974550    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:06.075180 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:06.075158    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:06.514425 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:06.514391    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:06.567571 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:06.567541    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:07.951038 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:07.951006    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:08.051778 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:08.051733    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:08.732965 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:08.732928    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:08.833656 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:08.833630    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:09.099584 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:09.099548    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:09.203032 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:09.203004    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:19.080672 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.080639    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:19.246671 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.246631    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5"]
Jul 29 23:52:19.247206 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.247187    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a51c264-3fe9-41fc-a684-93ae06c45a1c" containerName="main"
Jul 29 23:52:19.247206 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.247208    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a51c264-3fe9-41fc-a684-93ae06c45a1c" containerName="main"
Jul 29 23:52:19.247372 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.247316    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a51c264-3fe9-41fc-a684-93ae06c45a1c" containerName="main"
Jul 29 23:52:19.251682 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.251666    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5"
Jul 29 23:52:19.255884 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.255856    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 29 23:52:19.256044 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.256028    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-4nhtv\""
Jul 29 23:52:19.256137 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.256116    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 29 23:52:19.256279 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.256255    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 29 23:52:19.256409 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.256281    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-gc9wb\""
Jul 29 23:52:19.257979 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.257953    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5"]
Jul 29 23:52:19.409260 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.409231    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6zlt\" (UniqueName: \"kubernetes.io/projected/27e07296-8123-4e4e-a147-23c327a9306b-kube-api-access-s6zlt\") pod \"stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5\" (UID: \"27e07296-8123-4e4e-a147-23c327a9306b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5"
Jul 29 23:52:19.409419 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.409271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27e07296-8123-4e4e-a147-23c327a9306b-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5\" (UID: \"27e07296-8123-4e4e-a147-23c327a9306b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5"
Jul 29 23:52:19.509775 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.509746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6zlt\" (UniqueName: \"kubernetes.io/projected/27e07296-8123-4e4e-a147-23c327a9306b-kube-api-access-s6zlt\") pod \"stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5\" (UID: \"27e07296-8123-4e4e-a147-23c327a9306b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5"
Jul 29 23:52:19.509943 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.509788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27e07296-8123-4e4e-a147-23c327a9306b-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5\" (UID: \"27e07296-8123-4e4e-a147-23c327a9306b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5"
Jul 29 23:52:19.512230 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.512205    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27e07296-8123-4e4e-a147-23c327a9306b-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5\" (UID: \"27e07296-8123-4e4e-a147-23c327a9306b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5"
Jul 29 23:52:19.517532 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.517507    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6zlt\" (UniqueName: \"kubernetes.io/projected/27e07296-8123-4e4e-a147-23c327a9306b-kube-api-access-s6zlt\") pod \"stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5\" (UID: \"27e07296-8123-4e4e-a147-23c327a9306b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5"
Jul 29 23:52:19.563495 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.563469    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5"
Jul 29 23:52:19.696071 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.696046    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5"]
Jul 29 23:52:19.698506 ip-10-0-128-242 kubenswrapper[2581]: W0729 23:52:19.698463    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e07296_8123_4e4e_a147_23c327a9306b.slice/crio-cce219224e95449280695c37fd2592bfcaaa29cb952208aeb06dcdebd770659f WatchSource:0}: Error finding container cce219224e95449280695c37fd2592bfcaaa29cb952208aeb06dcdebd770659f: Status 404 returned error can't find the container with id cce219224e95449280695c37fd2592bfcaaa29cb952208aeb06dcdebd770659f
Jul 29 23:52:19.992548 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:19.992463    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:20.576654 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:20.576617    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5" event={"ID":"27e07296-8123-4e4e-a147-23c327a9306b","Type":"ContainerStarted","Data":"92c8d45e807636721e37e1ec1d460ee7bcc6b092d431e9a8511aebe118da597c"}
Jul 29 23:52:20.577123 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:20.576674    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5" event={"ID":"27e07296-8123-4e4e-a147-23c327a9306b","Type":"ContainerStarted","Data":"cce219224e95449280695c37fd2592bfcaaa29cb952208aeb06dcdebd770659f"}
Jul 29 23:52:20.577123 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:20.576736    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5"
Jul 29 23:52:20.593910 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:20.593866    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5" podStartSLOduration=1.5938522979999998 podStartE2EDuration="1.593852298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:52:20.592207539 +0000 UTC m=+3963.150552357" watchObservedRunningTime="2026-07-29 23:52:20.593852298 +0000 UTC m=+3963.152197113"
Jul 29 23:52:31.065530 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:52:31.065488    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 29 23:52:31.625815 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:31.625764    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:52:49.094618 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:49.094570    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:52:51.582493 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:52:51.582460    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5"
Jul 29 23:54:34.627047 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:54:34.627000    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 29 23:54:34.807450 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:54:34.807412    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:54:35.186497 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:54:35.186467    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:55:09.416820 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:55:09.416764    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:55:12.947101 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:55:12.947059    2581 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5" secret="" err="secret \"stop-feature-test-epp-sa-dockercfg-4nhtv\" not found"
Jul 29 23:55:13.018586 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:55:13.018420    2581 secret.go:189] Couldn't get secret e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-self-signed-certs: secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 29 23:55:13.018586 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:55:13.018521    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27e07296-8123-4e4e-a147-23c327a9306b-tls-certs podName:27e07296-8123-4e4e-a147-23c327a9306b nodeName:}" failed. No retries permitted until 2026-07-29 23:55:13.518497339 +0000 UTC m=+4136.076842143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/27e07296-8123-4e4e-a147-23c327a9306b-tls-certs") pod "stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5" (UID: "27e07296-8123-4e4e-a147-23c327a9306b") : secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 29 23:55:13.523077 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:55:13.523049    2581 secret.go:189] Couldn't get secret e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-self-signed-certs: secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 29 23:55:13.523242 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:55:13.523126    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27e07296-8123-4e4e-a147-23c327a9306b-tls-certs podName:27e07296-8123-4e4e-a147-23c327a9306b nodeName:}" failed. No retries permitted until 2026-07-29 23:55:14.523108155 +0000 UTC m=+4137.081452951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/27e07296-8123-4e4e-a147-23c327a9306b-tls-certs") pod "stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5" (UID: "27e07296-8123-4e4e-a147-23c327a9306b") : secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 29 23:55:14.227781 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:55:14.227737    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:55:20.555396 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:55:20.555362    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:56:31.947437 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:56:31.947400    2581 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5" secret="" err="secret \"stop-feature-test-epp-sa-dockercfg-4nhtv\" not found"
Jul 29 23:56:38.187270 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:56:38.187230    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 29 23:56:38.771234 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:56:38.771203    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:57:36.641403 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:36.641364    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5"]
Jul 29 23:57:36.682827 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:36.682785    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:57:38.154014 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:38.153966    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:57:41.582439 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:41.582399    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5" podUID="27e07296-8123-4e4e-a147-23c327a9306b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:57:47.762891 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:47.762850    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:57:47.814840 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:47.814814    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:57:51.582607 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:51.582570    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5" podUID="27e07296-8123-4e4e-a147-23c327a9306b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:57:51.665883 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:51.665852    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5" podUID="27e07296-8123-4e4e-a147-23c327a9306b" containerName="main" containerID="cri-o://92c8d45e807636721e37e1ec1d460ee7bcc6b092d431e9a8511aebe118da597c" gracePeriod=45
Jul 29 23:57:51.910682 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:51.910662    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5"
Jul 29 23:57:51.946538 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:51.946507    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6zlt\" (UniqueName: \"kubernetes.io/projected/27e07296-8123-4e4e-a147-23c327a9306b-kube-api-access-s6zlt\") pod \"27e07296-8123-4e4e-a147-23c327a9306b\" (UID: \"27e07296-8123-4e4e-a147-23c327a9306b\") "
Jul 29 23:57:51.946664 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:51.946571    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27e07296-8123-4e4e-a147-23c327a9306b-tls-certs\") pod \"27e07296-8123-4e4e-a147-23c327a9306b\" (UID: \"27e07296-8123-4e4e-a147-23c327a9306b\") "
Jul 29 23:57:51.948833 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:51.948786    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e07296-8123-4e4e-a147-23c327a9306b-kube-api-access-s6zlt" (OuterVolumeSpecName: "kube-api-access-s6zlt") pod "27e07296-8123-4e4e-a147-23c327a9306b" (UID: "27e07296-8123-4e4e-a147-23c327a9306b"). InnerVolumeSpecName "kube-api-access-s6zlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:57:51.948947 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:51.948829    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27e07296-8123-4e4e-a147-23c327a9306b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27e07296-8123-4e4e-a147-23c327a9306b" (UID: "27e07296-8123-4e4e-a147-23c327a9306b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:57:52.047882 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:52.047854    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27e07296-8123-4e4e-a147-23c327a9306b-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:57:52.047882 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:52.047880    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6zlt\" (UniqueName: \"kubernetes.io/projected/27e07296-8123-4e4e-a147-23c327a9306b-kube-api-access-s6zlt\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 29 23:57:52.103636 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:52.103567    2581 generic.go:356] "Generic (PLEG): container finished" podID="27e07296-8123-4e4e-a147-23c327a9306b" containerID="92c8d45e807636721e37e1ec1d460ee7bcc6b092d431e9a8511aebe118da597c" exitCode=0
Jul 29 23:57:52.103636 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:52.103613    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5"
Jul 29 23:57:52.103829 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:52.103653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5" event={"ID":"27e07296-8123-4e4e-a147-23c327a9306b","Type":"ContainerDied","Data":"92c8d45e807636721e37e1ec1d460ee7bcc6b092d431e9a8511aebe118da597c"}
Jul 29 23:57:52.103829 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:52.103705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5" event={"ID":"27e07296-8123-4e4e-a147-23c327a9306b","Type":"ContainerDied","Data":"cce219224e95449280695c37fd2592bfcaaa29cb952208aeb06dcdebd770659f"}
Jul 29 23:57:52.103829 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:52.103727    2581 scope.go:117] "RemoveContainer" containerID="92c8d45e807636721e37e1ec1d460ee7bcc6b092d431e9a8511aebe118da597c"
Jul 29 23:57:52.113740 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:52.113559    2581 scope.go:117] "RemoveContainer" containerID="92c8d45e807636721e37e1ec1d460ee7bcc6b092d431e9a8511aebe118da597c"
Jul 29 23:57:52.114156 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:57:52.114136    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c8d45e807636721e37e1ec1d460ee7bcc6b092d431e9a8511aebe118da597c\": container with ID starting with 92c8d45e807636721e37e1ec1d460ee7bcc6b092d431e9a8511aebe118da597c not found: ID does not exist" containerID="92c8d45e807636721e37e1ec1d460ee7bcc6b092d431e9a8511aebe118da597c"
Jul 29 23:57:52.114232 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:52.114164    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c8d45e807636721e37e1ec1d460ee7bcc6b092d431e9a8511aebe118da597c"} err="failed to get container status \"92c8d45e807636721e37e1ec1d460ee7bcc6b092d431e9a8511aebe118da597c\": rpc error: code = NotFound desc = could not find container \"92c8d45e807636721e37e1ec1d460ee7bcc6b092d431e9a8511aebe118da597c\": container with ID starting with 92c8d45e807636721e37e1ec1d460ee7bcc6b092d431e9a8511aebe118da597c not found: ID does not exist"
Jul 29 23:57:52.128884 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:52.128863    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5"]
Jul 29 23:57:52.133943 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:52.133923    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bz9m5"]
Jul 29 23:57:53.951703 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:53.951673    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27e07296-8123-4e4e-a147-23c327a9306b" path="/var/lib/kubelet/pods/27e07296-8123-4e4e-a147-23c327a9306b/volumes"
Jul 29 23:57:56.572261 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:56.572230    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:57:56.674837 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:56.674785    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:57:56.733104 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:57:56.733074    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:58:02.473726 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:58:02.473690    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:58:08.053227 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:58:08.053189    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:58:08.146489 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:58:08.146463    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:58:15.026466 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:58:15.026378    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 29 23:58:41.771961 ip-10-0-128-242 kubenswrapper[2581]: E0729 23:58:41.771917    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 29 23:58:42.330418 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:58:42.330387    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 29 23:59:44.833561 ip-10-0-128-242 kubenswrapper[2581]: I0729 23:59:44.833525    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:00:33.235179 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:00:33.235148    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:00:45.331678 ip-10-0-128-242 kubenswrapper[2581]: E0730 00:00:45.331628    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 30 00:00:45.904744 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:00:45.904712    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 30 00:00:51.215041 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:00:51.214992    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:00:51.410192 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:00:51.410146    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:00:51.510399 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:00:51.510322    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:00:57.019535 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:00:57.019493    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:00:57.080968 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:00:57.080929    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:01:32.229709 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:01:32.229629    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:02:48.905925 ip-10-0-128-242 kubenswrapper[2581]: E0730 00:02:48.905888    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 30 00:02:49.467904 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:02:49.467875    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 30 00:04:09.590241 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:09.590154    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:04:26.811842 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:26.811788    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:04:26.947868 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:26.947833    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:04:27.047250 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:27.047210    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:04:32.763619 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:32.763583    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:04:32.864088 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:32.864046    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:04:38.153228 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.153198    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:04:38.439006 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.438928    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9"]
Jul 30 00:04:38.439328 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.439306    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27e07296-8123-4e4e-a147-23c327a9306b" containerName="main"
Jul 30 00:04:38.439328 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.439325    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="27e07296-8123-4e4e-a147-23c327a9306b" containerName="main"
Jul 30 00:04:38.439483 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.439395    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="27e07296-8123-4e4e-a147-23c327a9306b" containerName="main"
Jul 30 00:04:38.442212 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.442196    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9"
Jul 30 00:04:38.446439 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.446415    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-5nrzc\""
Jul 30 00:04:38.446568 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.446442    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Jul 30 00:04:38.446568 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.446469    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Jul 30 00:04:38.446568 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.446480    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-4gdvp\""
Jul 30 00:04:38.446568 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.446480    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 30 00:04:38.450204 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.450185    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9"]
Jul 30 00:04:38.466091 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.466069    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:04:38.582818 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.582768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4da1943e-2735-4c15-a859-b9929b6f506b-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9\" (UID: \"4da1943e-2735-4c15-a859-b9929b6f506b\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9"
Jul 30 00:04:38.582961 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.582838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp5l9\" (UniqueName: \"kubernetes.io/projected/4da1943e-2735-4c15-a859-b9929b6f506b-kube-api-access-kp5l9\") pod \"tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9\" (UID: \"4da1943e-2735-4c15-a859-b9929b6f506b\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9"
Jul 30 00:04:38.684162 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.684136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4da1943e-2735-4c15-a859-b9929b6f506b-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9\" (UID: \"4da1943e-2735-4c15-a859-b9929b6f506b\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9"
Jul 30 00:04:38.684281 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.684173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp5l9\" (UniqueName: \"kubernetes.io/projected/4da1943e-2735-4c15-a859-b9929b6f506b-kube-api-access-kp5l9\") pod \"tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9\" (UID: \"4da1943e-2735-4c15-a859-b9929b6f506b\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9"
Jul 30 00:04:38.686520 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.686491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4da1943e-2735-4c15-a859-b9929b6f506b-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9\" (UID: \"4da1943e-2735-4c15-a859-b9929b6f506b\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9"
Jul 30 00:04:38.692281 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.692224    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp5l9\" (UniqueName: \"kubernetes.io/projected/4da1943e-2735-4c15-a859-b9929b6f506b-kube-api-access-kp5l9\") pod \"tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9\" (UID: \"4da1943e-2735-4c15-a859-b9929b6f506b\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9"
Jul 30 00:04:38.752887 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.752855    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9"
Jul 30 00:04:38.880977 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.880946    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9"]
Jul 30 00:04:38.882656 ip-10-0-128-242 kubenswrapper[2581]: W0730 00:04:38.882621    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da1943e_2735_4c15_a859_b9929b6f506b.slice/crio-475db87c9be0ae13fd8533e886296551b95720cfb23a29148b69732cfb161bd7 WatchSource:0}: Error finding container 475db87c9be0ae13fd8533e886296551b95720cfb23a29148b69732cfb161bd7: Status 404 returned error can't find the container with id 475db87c9be0ae13fd8533e886296551b95720cfb23a29148b69732cfb161bd7
Jul 30 00:04:38.970752 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.970658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9" event={"ID":"4da1943e-2735-4c15-a859-b9929b6f506b","Type":"ContainerStarted","Data":"3c0beed18e7612ad3613fe73a3ee4f7becda04cf64cc53b36c2cf362cf741191"}
Jul 30 00:04:38.970752 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.970719    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9" event={"ID":"4da1943e-2735-4c15-a859-b9929b6f506b","Type":"ContainerStarted","Data":"475db87c9be0ae13fd8533e886296551b95720cfb23a29148b69732cfb161bd7"}
Jul 30 00:04:38.970969 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.970825    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9"
Jul 30 00:04:38.987071 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:38.987024    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9" podStartSLOduration=0.987009684 podStartE2EDuration="987.009684ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:04:38.986962354 +0000 UTC m=+4701.545307172" watchObservedRunningTime="2026-07-30 00:04:38.987009684 +0000 UTC m=+4701.545354500"
Jul 30 00:04:52.468540 ip-10-0-128-242 kubenswrapper[2581]: E0730 00:04:52.468497    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 30 00:04:53.032316 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:04:53.032283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 30 00:05:09.975656 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:05:09.975628    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9"
Jul 30 00:05:14.771740 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:05:14.771701    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:05:35.087253 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:05:35.087213    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:05:55.244764 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:05:55.244726    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:05:55.346278 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:05:55.346244    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:06:50.590159 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:06:50.590119    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9"]
Jul 30 00:06:50.603369 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:06:50.603336    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:06:50.704676 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:06:50.704645    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:06:56.033578 ip-10-0-128-242 kubenswrapper[2581]: E0730 00:06:56.033542    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 30 00:06:56.589933 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:06:56.589900    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 30 00:06:59.976248 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:06:59.976208    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9" podUID="4da1943e-2735-4c15-a859-b9929b6f506b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 30 00:07:05.610429 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:05.610367    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9" podUID="4da1943e-2735-4c15-a859-b9929b6f506b" containerName="main" containerID="cri-o://3c0beed18e7612ad3613fe73a3ee4f7becda04cf64cc53b36c2cf362cf741191" gracePeriod=45
Jul 30 00:07:05.859769 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:05.859747    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9"
Jul 30 00:07:05.914994 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:05.914926    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4da1943e-2735-4c15-a859-b9929b6f506b-tls-certs\") pod \"4da1943e-2735-4c15-a859-b9929b6f506b\" (UID: \"4da1943e-2735-4c15-a859-b9929b6f506b\") "
Jul 30 00:07:05.914994 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:05.914960    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp5l9\" (UniqueName: \"kubernetes.io/projected/4da1943e-2735-4c15-a859-b9929b6f506b-kube-api-access-kp5l9\") pod \"4da1943e-2735-4c15-a859-b9929b6f506b\" (UID: \"4da1943e-2735-4c15-a859-b9929b6f506b\") "
Jul 30 00:07:05.917113 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:05.917082    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4da1943e-2735-4c15-a859-b9929b6f506b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4da1943e-2735-4c15-a859-b9929b6f506b" (UID: "4da1943e-2735-4c15-a859-b9929b6f506b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:07:05.917240 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:05.917219    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4da1943e-2735-4c15-a859-b9929b6f506b-kube-api-access-kp5l9" (OuterVolumeSpecName: "kube-api-access-kp5l9") pod "4da1943e-2735-4c15-a859-b9929b6f506b" (UID: "4da1943e-2735-4c15-a859-b9929b6f506b"). InnerVolumeSpecName "kube-api-access-kp5l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:07:06.016538 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:06.016510    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4da1943e-2735-4c15-a859-b9929b6f506b-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 30 00:07:06.016538 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:06.016533    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp5l9\" (UniqueName: \"kubernetes.io/projected/4da1943e-2735-4c15-a859-b9929b6f506b-kube-api-access-kp5l9\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 30 00:07:06.636352 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:06.636310    2581 generic.go:356] "Generic (PLEG): container finished" podID="4da1943e-2735-4c15-a859-b9929b6f506b" containerID="3c0beed18e7612ad3613fe73a3ee4f7becda04cf64cc53b36c2cf362cf741191" exitCode=0
Jul 30 00:07:06.636785 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:06.636358    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9"
Jul 30 00:07:06.636785 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:06.636392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9" event={"ID":"4da1943e-2735-4c15-a859-b9929b6f506b","Type":"ContainerDied","Data":"3c0beed18e7612ad3613fe73a3ee4f7becda04cf64cc53b36c2cf362cf741191"}
Jul 30 00:07:06.636785 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:06.636450    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9" event={"ID":"4da1943e-2735-4c15-a859-b9929b6f506b","Type":"ContainerDied","Data":"475db87c9be0ae13fd8533e886296551b95720cfb23a29148b69732cfb161bd7"}
Jul 30 00:07:06.636785 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:06.636473    2581 scope.go:117] "RemoveContainer" containerID="3c0beed18e7612ad3613fe73a3ee4f7becda04cf64cc53b36c2cf362cf741191"
Jul 30 00:07:06.645167 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:06.645150    2581 scope.go:117] "RemoveContainer" containerID="3c0beed18e7612ad3613fe73a3ee4f7becda04cf64cc53b36c2cf362cf741191"
Jul 30 00:07:06.645429 ip-10-0-128-242 kubenswrapper[2581]: E0730 00:07:06.645407    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c0beed18e7612ad3613fe73a3ee4f7becda04cf64cc53b36c2cf362cf741191\": container with ID starting with 3c0beed18e7612ad3613fe73a3ee4f7becda04cf64cc53b36c2cf362cf741191 not found: ID does not exist" containerID="3c0beed18e7612ad3613fe73a3ee4f7becda04cf64cc53b36c2cf362cf741191"
Jul 30 00:07:06.645480 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:06.645441    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c0beed18e7612ad3613fe73a3ee4f7becda04cf64cc53b36c2cf362cf741191"} err="failed to get container status \"3c0beed18e7612ad3613fe73a3ee4f7becda04cf64cc53b36c2cf362cf741191\": rpc error: code = NotFound desc = could not find container \"3c0beed18e7612ad3613fe73a3ee4f7becda04cf64cc53b36c2cf362cf741191\": container with ID starting with 3c0beed18e7612ad3613fe73a3ee4f7becda04cf64cc53b36c2cf362cf741191 not found: ID does not exist"
Jul 30 00:07:06.652498 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:06.652471    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9"]
Jul 30 00:07:06.654858 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:06.654837    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-cb69fc99f-5zws9"]
Jul 30 00:07:07.952575 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:07.952543    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4da1943e-2735-4c15-a859-b9929b6f506b" path="/var/lib/kubelet/pods/4da1943e-2735-4c15-a859-b9929b6f506b/volumes"
Jul 30 00:07:11.342208 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:11.342116    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:07:11.439732 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:11.439701    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:07:17.201197 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:17.201160    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:07:18.333648 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.333613    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:07:18.499090 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.499064    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz"]
Jul 30 00:07:18.499482 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.499466    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4da1943e-2735-4c15-a859-b9929b6f506b" containerName="main"
Jul 30 00:07:18.499572 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.499485    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4da1943e-2735-4c15-a859-b9929b6f506b" containerName="main"
Jul 30 00:07:18.499630 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.499585    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4da1943e-2735-4c15-a859-b9929b6f506b" containerName="main"
Jul 30 00:07:18.503791 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.503773    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz"
Jul 30 00:07:18.508390 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.508367    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 30 00:07:18.508632 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.508609    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 30 00:07:18.508774 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.508753    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-8ltz7\""
Jul 30 00:07:18.508929 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.508911    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-cpsfq\""
Jul 30 00:07:18.509008 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.508994    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 30 00:07:18.511085 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.511056    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz"]
Jul 30 00:07:18.527545 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.527521    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:07:18.627281 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.627212    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c7a8de-4112-4fd7-9af7-858f9047fe98-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz\" (UID: \"d5c7a8de-4112-4fd7-9af7-858f9047fe98\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz"
Jul 30 00:07:18.627281 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.627254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbt4l\" (UniqueName: \"kubernetes.io/projected/d5c7a8de-4112-4fd7-9af7-858f9047fe98-kube-api-access-pbt4l\") pod \"prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz\" (UID: \"d5c7a8de-4112-4fd7-9af7-858f9047fe98\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz"
Jul 30 00:07:18.727991 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.727962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c7a8de-4112-4fd7-9af7-858f9047fe98-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz\" (UID: \"d5c7a8de-4112-4fd7-9af7-858f9047fe98\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz"
Jul 30 00:07:18.728132 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.728009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbt4l\" (UniqueName: \"kubernetes.io/projected/d5c7a8de-4112-4fd7-9af7-858f9047fe98-kube-api-access-pbt4l\") pod \"prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz\" (UID: \"d5c7a8de-4112-4fd7-9af7-858f9047fe98\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz"
Jul 30 00:07:18.730338 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.730316    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c7a8de-4112-4fd7-9af7-858f9047fe98-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz\" (UID: \"d5c7a8de-4112-4fd7-9af7-858f9047fe98\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz"
Jul 30 00:07:18.737069 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.737046    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbt4l\" (UniqueName: \"kubernetes.io/projected/d5c7a8de-4112-4fd7-9af7-858f9047fe98-kube-api-access-pbt4l\") pod \"prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz\" (UID: \"d5c7a8de-4112-4fd7-9af7-858f9047fe98\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz"
Jul 30 00:07:18.822227 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.822202    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz"
Jul 30 00:07:18.952242 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:18.952219    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz"]
Jul 30 00:07:18.954254 ip-10-0-128-242 kubenswrapper[2581]: W0730 00:07:18.954225    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c7a8de_4112_4fd7_9af7_858f9047fe98.slice/crio-af1ee99ee31ff0711c09ba3676d553a120e8cb01497b92315e810f6060e66ab8 WatchSource:0}: Error finding container af1ee99ee31ff0711c09ba3676d553a120e8cb01497b92315e810f6060e66ab8: Status 404 returned error can't find the container with id af1ee99ee31ff0711c09ba3676d553a120e8cb01497b92315e810f6060e66ab8
Jul 30 00:07:19.697958 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:19.697908    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz" event={"ID":"d5c7a8de-4112-4fd7-9af7-858f9047fe98","Type":"ContainerStarted","Data":"92e22623577548753496016adb18dadb0a76fc7efd8b5294cc0e007f869f45f1"}
Jul 30 00:07:19.698429 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:19.697973    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz" event={"ID":"d5c7a8de-4112-4fd7-9af7-858f9047fe98","Type":"ContainerStarted","Data":"af1ee99ee31ff0711c09ba3676d553a120e8cb01497b92315e810f6060e66ab8"}
Jul 30 00:07:19.698429 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:19.698089    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz"
Jul 30 00:07:19.714457 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:19.714413    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz" podStartSLOduration=1.7143984410000002 podStartE2EDuration="1.714398441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:07:19.713282454 +0000 UTC m=+4862.271627271" watchObservedRunningTime="2026-07-30 00:07:19.714398441 +0000 UTC m=+4862.272743257"
Jul 30 00:07:22.636855 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:22.636822    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:07:22.697292 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:22.697263    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:07:50.704236 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:07:50.704209    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz"
Jul 30 00:08:02.161788 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:08:02.161749    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:08:59.590631 ip-10-0-128-242 kubenswrapper[2581]: E0730 00:08:59.590588    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 30 00:09:00.143238 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:09:00.143210    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 30 00:10:14.165496 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:14.165412    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz"]
Jul 30 00:10:14.191641 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:14.191616    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:10:15.736658 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:15.736627    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:10:20.703990 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:20.703953    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz" podUID="d5c7a8de-4112-4fd7-9af7-858f9047fe98" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 30 00:10:29.184485 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:29.184442    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz" podUID="d5c7a8de-4112-4fd7-9af7-858f9047fe98" containerName="main" containerID="cri-o://92e22623577548753496016adb18dadb0a76fc7efd8b5294cc0e007f869f45f1" gracePeriod=45
Jul 30 00:10:29.438906 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:29.438848    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz"
Jul 30 00:10:29.545906 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:29.545875    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c7a8de-4112-4fd7-9af7-858f9047fe98-tls-certs\") pod \"d5c7a8de-4112-4fd7-9af7-858f9047fe98\" (UID: \"d5c7a8de-4112-4fd7-9af7-858f9047fe98\") "
Jul 30 00:10:29.546084 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:29.545945    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbt4l\" (UniqueName: \"kubernetes.io/projected/d5c7a8de-4112-4fd7-9af7-858f9047fe98-kube-api-access-pbt4l\") pod \"d5c7a8de-4112-4fd7-9af7-858f9047fe98\" (UID: \"d5c7a8de-4112-4fd7-9af7-858f9047fe98\") "
Jul 30 00:10:29.548188 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:29.548157    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c7a8de-4112-4fd7-9af7-858f9047fe98-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5c7a8de-4112-4fd7-9af7-858f9047fe98" (UID: "d5c7a8de-4112-4fd7-9af7-858f9047fe98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:10:29.548305 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:29.548250    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c7a8de-4112-4fd7-9af7-858f9047fe98-kube-api-access-pbt4l" (OuterVolumeSpecName: "kube-api-access-pbt4l") pod "d5c7a8de-4112-4fd7-9af7-858f9047fe98" (UID: "d5c7a8de-4112-4fd7-9af7-858f9047fe98"). InnerVolumeSpecName "kube-api-access-pbt4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:10:29.554362 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:29.554342    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5c7a8de-4112-4fd7-9af7-858f9047fe98" containerID="92e22623577548753496016adb18dadb0a76fc7efd8b5294cc0e007f869f45f1" exitCode=0
Jul 30 00:10:29.554417 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:29.554396    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz"
Jul 30 00:10:29.554453 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:29.554421    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz" event={"ID":"d5c7a8de-4112-4fd7-9af7-858f9047fe98","Type":"ContainerDied","Data":"92e22623577548753496016adb18dadb0a76fc7efd8b5294cc0e007f869f45f1"}
Jul 30 00:10:29.554488 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:29.554472    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz" event={"ID":"d5c7a8de-4112-4fd7-9af7-858f9047fe98","Type":"ContainerDied","Data":"af1ee99ee31ff0711c09ba3676d553a120e8cb01497b92315e810f6060e66ab8"}
Jul 30 00:10:29.554563 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:29.554503    2581 scope.go:117] "RemoveContainer" containerID="92e22623577548753496016adb18dadb0a76fc7efd8b5294cc0e007f869f45f1"
Jul 30 00:10:29.563433 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:29.563417    2581 scope.go:117] "RemoveContainer" containerID="92e22623577548753496016adb18dadb0a76fc7efd8b5294cc0e007f869f45f1"
Jul 30 00:10:29.563676 ip-10-0-128-242 kubenswrapper[2581]: E0730 00:10:29.563657    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92e22623577548753496016adb18dadb0a76fc7efd8b5294cc0e007f869f45f1\": container with ID starting with 92e22623577548753496016adb18dadb0a76fc7efd8b5294cc0e007f869f45f1 not found: ID does not exist" containerID="92e22623577548753496016adb18dadb0a76fc7efd8b5294cc0e007f869f45f1"
Jul 30 00:10:29.563742 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:29.563684    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92e22623577548753496016adb18dadb0a76fc7efd8b5294cc0e007f869f45f1"} err="failed to get container status \"92e22623577548753496016adb18dadb0a76fc7efd8b5294cc0e007f869f45f1\": rpc error: code = NotFound desc = could not find container \"92e22623577548753496016adb18dadb0a76fc7efd8b5294cc0e007f869f45f1\": container with ID starting with 92e22623577548753496016adb18dadb0a76fc7efd8b5294cc0e007f869f45f1 not found: ID does not exist"
Jul 30 00:10:29.576732 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:29.576712    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz"]
Jul 30 00:10:29.578275 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:29.578252    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-85fc59f59-dvdbz"]
Jul 30 00:10:29.646668 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:29.646641    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c7a8de-4112-4fd7-9af7-858f9047fe98-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 30 00:10:29.646668 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:29.646668    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbt4l\" (UniqueName: \"kubernetes.io/projected/d5c7a8de-4112-4fd7-9af7-858f9047fe98-kube-api-access-pbt4l\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 30 00:10:29.952778 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:29.952748    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5c7a8de-4112-4fd7-9af7-858f9047fe98" path="/var/lib/kubelet/pods/d5c7a8de-4112-4fd7-9af7-858f9047fe98/volumes"
Jul 30 00:10:37.283500 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:37.283447    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:10:37.400826 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:37.400782    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:10:42.297069 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:42.297033    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:10:47.709667 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:47.709635    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:10:47.992658 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:47.992566    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf"]
Jul 30 00:10:47.992945 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:47.992931    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c7a8de-4112-4fd7-9af7-858f9047fe98" containerName="main"
Jul 30 00:10:47.992945 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:47.992944    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c7a8de-4112-4fd7-9af7-858f9047fe98" containerName="main"
Jul 30 00:10:47.993092 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:47.993010    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c7a8de-4112-4fd7-9af7-858f9047fe98" containerName="main"
Jul 30 00:10:47.995863 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:47.995841    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf"
Jul 30 00:10:48.000069 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.000044    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-g7lfm\""
Jul 30 00:10:48.000187 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.000093    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-2cct7\""
Jul 30 00:10:48.000187 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.000043    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Jul 30 00:10:48.000187 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.000096    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 30 00:10:48.000416 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.000398    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Jul 30 00:10:48.005361 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.005338    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf"]
Jul 30 00:10:48.031678 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.031654    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:10:48.103319 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.103292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtwp6\" (UniqueName: \"kubernetes.io/projected/32fa3dc4-9211-4789-a909-49c93dc6a8dc-kube-api-access-gtwp6\") pod \"rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf\" (UID: \"32fa3dc4-9211-4789-a909-49c93dc6a8dc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf"
Jul 30 00:10:48.103489 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.103409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32fa3dc4-9211-4789-a909-49c93dc6a8dc-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf\" (UID: \"32fa3dc4-9211-4789-a909-49c93dc6a8dc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf"
Jul 30 00:10:48.183625 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.183560    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:10:48.204225 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.204199    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32fa3dc4-9211-4789-a909-49c93dc6a8dc-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf\" (UID: \"32fa3dc4-9211-4789-a909-49c93dc6a8dc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf"
Jul 30 00:10:48.204360 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.204243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtwp6\" (UniqueName: \"kubernetes.io/projected/32fa3dc4-9211-4789-a909-49c93dc6a8dc-kube-api-access-gtwp6\") pod \"rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf\" (UID: \"32fa3dc4-9211-4789-a909-49c93dc6a8dc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf"
Jul 30 00:10:48.206641 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.206617    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32fa3dc4-9211-4789-a909-49c93dc6a8dc-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf\" (UID: \"32fa3dc4-9211-4789-a909-49c93dc6a8dc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf"
Jul 30 00:10:48.211877 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.211858    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtwp6\" (UniqueName: \"kubernetes.io/projected/32fa3dc4-9211-4789-a909-49c93dc6a8dc-kube-api-access-gtwp6\") pod \"rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf\" (UID: \"32fa3dc4-9211-4789-a909-49c93dc6a8dc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf"
Jul 30 00:10:48.283782 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.283699    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:10:48.307793 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.307768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf"
Jul 30 00:10:48.439154 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.439131    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf"]
Jul 30 00:10:48.441723 ip-10-0-128-242 kubenswrapper[2581]: W0730 00:10:48.441696    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fa3dc4_9211_4789_a909_49c93dc6a8dc.slice/crio-1f661751bc6f8056688431e86e56e257c07c828ce5504d77ae86874dd2eff4cd WatchSource:0}: Error finding container 1f661751bc6f8056688431e86e56e257c07c828ce5504d77ae86874dd2eff4cd: Status 404 returned error can't find the container with id 1f661751bc6f8056688431e86e56e257c07c828ce5504d77ae86874dd2eff4cd
Jul 30 00:10:48.644354 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.644319    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf" event={"ID":"32fa3dc4-9211-4789-a909-49c93dc6a8dc","Type":"ContainerStarted","Data":"41ea3ec17c961e91735158e45e96a53769ea51432342c97ca56089a41c240789"}
Jul 30 00:10:48.644529 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.644408    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf" event={"ID":"32fa3dc4-9211-4789-a909-49c93dc6a8dc","Type":"ContainerStarted","Data":"1f661751bc6f8056688431e86e56e257c07c828ce5504d77ae86874dd2eff4cd"}
Jul 30 00:10:48.644584 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.644536    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf"
Jul 30 00:10:48.660893 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:48.660843    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf" podStartSLOduration=1.660826725 podStartE2EDuration="1.660826725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:10:48.659447087 +0000 UTC m=+5071.217791903" watchObservedRunningTime="2026-07-30 00:10:48.660826725 +0000 UTC m=+5071.219171542"
Jul 30 00:10:58.401485 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:10:58.401448    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:11:03.143924 ip-10-0-128-242 kubenswrapper[2581]: E0730 00:11:03.143886    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 30 00:11:03.712783 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:11:03.712752    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 30 00:11:16.364331 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:11:16.364296    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:11:19.651166 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:11:19.651135    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf"
Jul 30 00:11:56.909237 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:11:56.909202    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf"]
Jul 30 00:11:57.172256 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:11:57.172171    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:11:57.468127 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:11:57.468046    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:11:59.651368 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:11:59.651324    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf" podUID="32fa3dc4-9211-4789-a909-49c93dc6a8dc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 30 00:12:09.651312 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:09.651276    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf" podUID="32fa3dc4-9211-4789-a909-49c93dc6a8dc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 30 00:12:11.928848 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:11.928785    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf" podUID="32fa3dc4-9211-4789-a909-49c93dc6a8dc" containerName="main" containerID="cri-o://41ea3ec17c961e91735158e45e96a53769ea51432342c97ca56089a41c240789" gracePeriod=45
Jul 30 00:12:12.184042 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:12.183988    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf"
Jul 30 00:12:12.218336 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:12.218313    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtwp6\" (UniqueName: \"kubernetes.io/projected/32fa3dc4-9211-4789-a909-49c93dc6a8dc-kube-api-access-gtwp6\") pod \"32fa3dc4-9211-4789-a909-49c93dc6a8dc\" (UID: \"32fa3dc4-9211-4789-a909-49c93dc6a8dc\") "
Jul 30 00:12:12.218512 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:12.218499    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32fa3dc4-9211-4789-a909-49c93dc6a8dc-tls-certs\") pod \"32fa3dc4-9211-4789-a909-49c93dc6a8dc\" (UID: \"32fa3dc4-9211-4789-a909-49c93dc6a8dc\") "
Jul 30 00:12:12.220377 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:12.220345    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32fa3dc4-9211-4789-a909-49c93dc6a8dc-kube-api-access-gtwp6" (OuterVolumeSpecName: "kube-api-access-gtwp6") pod "32fa3dc4-9211-4789-a909-49c93dc6a8dc" (UID: "32fa3dc4-9211-4789-a909-49c93dc6a8dc"). InnerVolumeSpecName "kube-api-access-gtwp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:12:12.220554 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:12.220531    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32fa3dc4-9211-4789-a909-49c93dc6a8dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32fa3dc4-9211-4789-a909-49c93dc6a8dc" (UID: "32fa3dc4-9211-4789-a909-49c93dc6a8dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:12:12.319273 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:12.319246    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32fa3dc4-9211-4789-a909-49c93dc6a8dc-tls-certs\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 30 00:12:12.319273 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:12.319271    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtwp6\" (UniqueName: \"kubernetes.io/projected/32fa3dc4-9211-4789-a909-49c93dc6a8dc-kube-api-access-gtwp6\") on node \"ip-10-0-128-242.ec2.internal\" DevicePath \"\""
Jul 30 00:12:13.029591 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:13.029557    2581 generic.go:356] "Generic (PLEG): container finished" podID="32fa3dc4-9211-4789-a909-49c93dc6a8dc" containerID="41ea3ec17c961e91735158e45e96a53769ea51432342c97ca56089a41c240789" exitCode=0
Jul 30 00:12:13.029591 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:13.029590    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf"
Jul 30 00:12:13.030100 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:13.029639    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf" event={"ID":"32fa3dc4-9211-4789-a909-49c93dc6a8dc","Type":"ContainerDied","Data":"41ea3ec17c961e91735158e45e96a53769ea51432342c97ca56089a41c240789"}
Jul 30 00:12:13.030100 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:13.029681    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf" event={"ID":"32fa3dc4-9211-4789-a909-49c93dc6a8dc","Type":"ContainerDied","Data":"1f661751bc6f8056688431e86e56e257c07c828ce5504d77ae86874dd2eff4cd"}
Jul 30 00:12:13.030100 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:13.029701    2581 scope.go:117] "RemoveContainer" containerID="41ea3ec17c961e91735158e45e96a53769ea51432342c97ca56089a41c240789"
Jul 30 00:12:13.040690 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:13.040668    2581 scope.go:117] "RemoveContainer" containerID="41ea3ec17c961e91735158e45e96a53769ea51432342c97ca56089a41c240789"
Jul 30 00:12:13.040948 ip-10-0-128-242 kubenswrapper[2581]: E0730 00:12:13.040929    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41ea3ec17c961e91735158e45e96a53769ea51432342c97ca56089a41c240789\": container with ID starting with 41ea3ec17c961e91735158e45e96a53769ea51432342c97ca56089a41c240789 not found: ID does not exist" containerID="41ea3ec17c961e91735158e45e96a53769ea51432342c97ca56089a41c240789"
Jul 30 00:12:13.041019 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:13.040960    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41ea3ec17c961e91735158e45e96a53769ea51432342c97ca56089a41c240789"} err="failed to get container status \"41ea3ec17c961e91735158e45e96a53769ea51432342c97ca56089a41c240789\": rpc error: code = NotFound desc = could not find container \"41ea3ec17c961e91735158e45e96a53769ea51432342c97ca56089a41c240789\": container with ID starting with 41ea3ec17c961e91735158e45e96a53769ea51432342c97ca56089a41c240789 not found: ID does not exist"
Jul 30 00:12:13.052784 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:13.052760    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf"]
Jul 30 00:12:13.056287 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:13.056269    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-56bdd7fbf-c6bjf"]
Jul 30 00:12:13.560855 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:13.560790    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:12:13.661013 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:13.660978    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:12:13.764217 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:13.764183    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:12:13.952591 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:13.952563    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32fa3dc4-9211-4789-a909-49c93dc6a8dc" path="/var/lib/kubelet/pods/32fa3dc4-9211-4789-a909-49c93dc6a8dc/volumes"
Jul 30 00:12:19.518020 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:19.517951    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:12:24.947107 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:24.947074    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:12:25.011447 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:12:25.011417    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:13:06.713442 ip-10-0-128-242 kubenswrapper[2581]: E0730 00:13:06.713400    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp" podUID="56094732-90fc-4ea7-b8cb-b551ca6cd72c"
Jul 30 00:13:07.272950 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:13:07.272918    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp"
Jul 30 00:13:32.652778 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:13:32.652708    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:13:33.832331 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:13:33.832300    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:13:33.933263 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:13:33.933237    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:13:38.218694 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:13:38.218661    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:13:38.280893 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:13:38.280849    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:14:00.893375 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:00.893340    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:01.120220 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:01.120187    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:01.980171 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:01.980138    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:02.356165 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:02.356136    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:03.633432 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:03.633396    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:14:03.642089 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:03.642069    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:03.913894 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:03.913810    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:04.719060 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:04.719026    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:04.933476 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:04.933446    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:05.746844 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:05.746815    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:05.965452 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:05.965425    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:06.779987 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:06.779948    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:06.989343 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:06.989315    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:07.776640 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:07.776605    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:07.987321 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:07.987295    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:08.777707 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:08.777672    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:08.989051 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:08.989024    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:09.793319 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:09.793287    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:10.008594 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:10.008563    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:10.786166 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:10.786134    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:11.026028 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:11.026000    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:11.824829 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:11.822417    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:12.072240 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:12.072182    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:12.864707 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:12.864677    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:13.073081 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:13.073043    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:13.911985 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:13.911934    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:14.126726 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:14.126696    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:14.926348 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:14.926314    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:15.141606 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:15.141559    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:15.966918 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:15.966889    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:16.181156 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:16.181125    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:16.977369 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:16.977340    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:17.200467 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:17.200435    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:17.998957 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:17.998891    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:18.208393 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:18.208352    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:19.060974 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:19.060936    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:19.271929 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:19.271890    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:20.068844 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:20.068813    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:20.280781 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:20.280747    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:21.081406 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:21.081371    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:21.297522 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:21.297492    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:22.126368 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:22.126338    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:22.350218 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:22.350183    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:23.189843 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:23.189814    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:23.404456 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:23.404425    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:24.214173 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:24.214139    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:24.437468 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:24.437437    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:25.245409 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:25.245375    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:25.474863 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:25.474838    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:26.263695 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:26.263665    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:26.485434 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:26.485406    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:28.433168 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:28.433138    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:14:28.617521 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:28.617488    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:14:28.723055 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:28.722972    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:14:31.388359 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:31.388327    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7pv5p_8687fc04-c1c6-4547-9787-dc94bfc700c3/global-pull-secret-syncer/0.log"
Jul 30 00:14:31.494261 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:31.494221    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log"
Jul 30 00:14:31.607098 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:31.607072    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-242.ec2.internal_82d8343817331f460e5ea044619fe935/haproxy/0.log"
Jul 30 00:14:34.156068 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:34.156037    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:14:34.255880 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:34.255854    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:14:37.251964 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:37.251883    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-gvscv_78f0b0f8-0335-49a3-88f4-d47d5801d011/authorino/0.log"
Jul 30 00:14:37.631974 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:37.631944    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-48g2h_30e566be-2326-4a4f-bf39-ef84cd8aaa97/manager/0.log"
Jul 30 00:14:38.799783 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:38.799752    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6dca323-830f-4221-837c-e4ddfb530361/alertmanager/0.log"
Jul 30 00:14:38.825659 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:38.825636    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6dca323-830f-4221-837c-e4ddfb530361/config-reloader/0.log"
Jul 30 00:14:38.849344 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:38.849324    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6dca323-830f-4221-837c-e4ddfb530361/kube-rbac-proxy-web/0.log"
Jul 30 00:14:38.873661 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:38.873639    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6dca323-830f-4221-837c-e4ddfb530361/kube-rbac-proxy/0.log"
Jul 30 00:14:38.898423 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:38.898389    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6dca323-830f-4221-837c-e4ddfb530361/kube-rbac-proxy-metric/0.log"
Jul 30 00:14:38.922098 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:38.922078    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6dca323-830f-4221-837c-e4ddfb530361/prom-label-proxy/0.log"
Jul 30 00:14:38.976738 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:38.976719    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6dca323-830f-4221-837c-e4ddfb530361/init-config-reloader/0.log"
Jul 30 00:14:39.382255 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:39.382183    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jrx9p_3907412a-9084-46b6-b25c-eaf316459f5e/node-exporter/0.log"
Jul 30 00:14:39.411432 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:39.411412    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jrx9p_3907412a-9084-46b6-b25c-eaf316459f5e/kube-rbac-proxy/0.log"
Jul 30 00:14:39.435708 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:39.435685    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jrx9p_3907412a-9084-46b6-b25c-eaf316459f5e/init-textfile/0.log"
Jul 30 00:14:39.555094 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:39.555018    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-6mt7w_94e94917-7911-4a16-a229-367716e81a26/kube-rbac-proxy-main/0.log"
Jul 30 00:14:39.582904 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:39.582874    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-6mt7w_94e94917-7911-4a16-a229-367716e81a26/kube-rbac-proxy-self/0.log"
Jul 30 00:14:39.604855 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:39.604830    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-6mt7w_94e94917-7911-4a16-a229-367716e81a26/openshift-state-metrics/0.log"
Jul 30 00:14:39.652632 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:39.652575    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4220a124-428a-4538-bb23-03a987e1a661/prometheus/0.log"
Jul 30 00:14:39.670424 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:39.670405    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4220a124-428a-4538-bb23-03a987e1a661/config-reloader/0.log"
Jul 30 00:14:39.693514 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:39.693498    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4220a124-428a-4538-bb23-03a987e1a661/thanos-sidecar/0.log"
Jul 30 00:14:39.720683 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:39.720663    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4220a124-428a-4538-bb23-03a987e1a661/kube-rbac-proxy-web/0.log"
Jul 30 00:14:39.746281 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:39.746264    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4220a124-428a-4538-bb23-03a987e1a661/kube-rbac-proxy/0.log"
Jul 30 00:14:39.769532 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:39.769503    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4220a124-428a-4538-bb23-03a987e1a661/kube-rbac-proxy-thanos/0.log"
Jul 30 00:14:39.794927 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:39.794909    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4220a124-428a-4538-bb23-03a987e1a661/init-config-reloader/0.log"
Jul 30 00:14:39.834033 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:39.834009    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-vq8rs_7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8/prometheus-operator/0.log"
Jul 30 00:14:39.854570 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:39.854552    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-vq8rs_7b8631d3-6cf9-4ca9-8c11-73dcb9b8b1d8/kube-rbac-proxy/0.log"
Jul 30 00:14:39.882376 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:39.882355    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-98g9k_0cbd4a90-8b07-483b-9fc6-e3ac61d55df9/prometheus-operator-admission-webhook/0.log"
Jul 30 00:14:40.088068 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:40.088044    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6dn6d_9a191d3f-9334-4881-8380-654a3b33d7d0/konnectivity-agent/0.log" limit=1024
Jul 30 00:14:41.104412 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:41.104381    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-sk5qv_2fe2a7d4-0e16-4f28-9173-d5765b53e33b/networking-console-plugin/0.log"
Jul 30 00:14:41.839287 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:41.839252    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"]
Jul 30 00:14:41.839700 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:41.839683    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32fa3dc4-9211-4789-a909-49c93dc6a8dc" containerName="main"
Jul 30 00:14:41.839811 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:41.839702    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="32fa3dc4-9211-4789-a909-49c93dc6a8dc" containerName="main"
Jul 30 00:14:41.839811 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:41.839793    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="32fa3dc4-9211-4789-a909-49c93dc6a8dc" containerName="main"
Jul 30 00:14:41.843033 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:41.843010    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:41.845895 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:41.845873    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6zt68\"/\"kube-root-ca.crt\""
Jul 30 00:14:41.846016 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:41.845878    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6zt68\"/\"openshift-service-ca.crt\""
Jul 30 00:14:41.847245 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:41.847226    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6zt68\"/\"default-dockercfg-k9xqk\""
Jul 30 00:14:41.851410 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:41.851388    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"]
Jul 30 00:14:41.944185 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:41.944145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11f77381-bc1d-427c-ac92-8f9fc8eb5261-lib-modules\") pod \"perf-node-gather-daemonset-vsx47\" (UID: \"11f77381-bc1d-427c-ac92-8f9fc8eb5261\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:41.944185 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:41.944190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11f77381-bc1d-427c-ac92-8f9fc8eb5261-sys\") pod \"perf-node-gather-daemonset-vsx47\" (UID: \"11f77381-bc1d-427c-ac92-8f9fc8eb5261\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:41.944402 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:41.944211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94bzp\" (UniqueName: \"kubernetes.io/projected/11f77381-bc1d-427c-ac92-8f9fc8eb5261-kube-api-access-94bzp\") pod \"perf-node-gather-daemonset-vsx47\" (UID: \"11f77381-bc1d-427c-ac92-8f9fc8eb5261\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:41.944402 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:41.944321    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11f77381-bc1d-427c-ac92-8f9fc8eb5261-podres\") pod \"perf-node-gather-daemonset-vsx47\" (UID: \"11f77381-bc1d-427c-ac92-8f9fc8eb5261\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:41.944402 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:41.944374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11f77381-bc1d-427c-ac92-8f9fc8eb5261-proc\") pod \"perf-node-gather-daemonset-vsx47\" (UID: \"11f77381-bc1d-427c-ac92-8f9fc8eb5261\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:42.045671 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:42.045636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11f77381-bc1d-427c-ac92-8f9fc8eb5261-podres\") pod \"perf-node-gather-daemonset-vsx47\" (UID: \"11f77381-bc1d-427c-ac92-8f9fc8eb5261\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:42.045854 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:42.045828    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11f77381-bc1d-427c-ac92-8f9fc8eb5261-podres\") pod \"perf-node-gather-daemonset-vsx47\" (UID: \"11f77381-bc1d-427c-ac92-8f9fc8eb5261\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:42.046062 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:42.046029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11f77381-bc1d-427c-ac92-8f9fc8eb5261-proc\") pod \"perf-node-gather-daemonset-vsx47\" (UID: \"11f77381-bc1d-427c-ac92-8f9fc8eb5261\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:42.046211 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:42.046095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11f77381-bc1d-427c-ac92-8f9fc8eb5261-lib-modules\") pod \"perf-node-gather-daemonset-vsx47\" (UID: \"11f77381-bc1d-427c-ac92-8f9fc8eb5261\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:42.046211 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:42.046108    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11f77381-bc1d-427c-ac92-8f9fc8eb5261-proc\") pod \"perf-node-gather-daemonset-vsx47\" (UID: \"11f77381-bc1d-427c-ac92-8f9fc8eb5261\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:42.046211 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:42.046182    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11f77381-bc1d-427c-ac92-8f9fc8eb5261-sys\") pod \"perf-node-gather-daemonset-vsx47\" (UID: \"11f77381-bc1d-427c-ac92-8f9fc8eb5261\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:42.046211 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:42.046202    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11f77381-bc1d-427c-ac92-8f9fc8eb5261-lib-modules\") pod \"perf-node-gather-daemonset-vsx47\" (UID: \"11f77381-bc1d-427c-ac92-8f9fc8eb5261\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:42.046444 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:42.046230    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11f77381-bc1d-427c-ac92-8f9fc8eb5261-sys\") pod \"perf-node-gather-daemonset-vsx47\" (UID: \"11f77381-bc1d-427c-ac92-8f9fc8eb5261\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:42.046444 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:42.046245    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94bzp\" (UniqueName: \"kubernetes.io/projected/11f77381-bc1d-427c-ac92-8f9fc8eb5261-kube-api-access-94bzp\") pod \"perf-node-gather-daemonset-vsx47\" (UID: \"11f77381-bc1d-427c-ac92-8f9fc8eb5261\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:42.053981 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:42.053961    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94bzp\" (UniqueName: \"kubernetes.io/projected/11f77381-bc1d-427c-ac92-8f9fc8eb5261-kube-api-access-94bzp\") pod \"perf-node-gather-daemonset-vsx47\" (UID: \"11f77381-bc1d-427c-ac92-8f9fc8eb5261\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:42.153473 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:42.153406    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:42.281763 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:42.281735    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"]
Jul 30 00:14:42.283558 ip-10-0-128-242 kubenswrapper[2581]: W0730 00:14:42.283524    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod11f77381_bc1d_427c_ac92_8f9fc8eb5261.slice/crio-123462925c4cd44085c65cae82ca2e934694fa55a328d0536c8b5ef3982d0719 WatchSource:0}: Error finding container 123462925c4cd44085c65cae82ca2e934694fa55a328d0536c8b5ef3982d0719: Status 404 returned error can't find the container with id 123462925c4cd44085c65cae82ca2e934694fa55a328d0536c8b5ef3982d0719
Jul 30 00:14:42.687430 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:42.687401    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-z9cvg_cdc35d2f-cfe4-4a6c-a75c-431cf95edc19/volume-data-source-validator/0.log"
Jul 30 00:14:42.701743 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:42.701715    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47" event={"ID":"11f77381-bc1d-427c-ac92-8f9fc8eb5261","Type":"ContainerStarted","Data":"cc1232f8f0f69d182bb7bf5b8f694409797704ca52e4e598351d62f4cbe02205"}
Jul 30 00:14:42.701905 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:42.701776    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:42.701905 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:42.701819    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47" event={"ID":"11f77381-bc1d-427c-ac92-8f9fc8eb5261","Type":"ContainerStarted","Data":"123462925c4cd44085c65cae82ca2e934694fa55a328d0536c8b5ef3982d0719"}
Jul 30 00:14:42.719211 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:42.719168    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47" podStartSLOduration=1.719153281 podStartE2EDuration="1.719153281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:14:42.717225599 +0000 UTC m=+5305.275570415" watchObservedRunningTime="2026-07-30 00:14:42.719153281 +0000 UTC m=+5305.277498096"
Jul 30 00:14:43.675446 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:43.675413    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-25gll_ab3523d2-8acf-462d-8458-3b2e6fa74f08/dns/0.log"
Jul 30 00:14:43.697225 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:43.697202    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-25gll_ab3523d2-8acf-462d-8458-3b2e6fa74f08/kube-rbac-proxy/0.log"
Jul 30 00:14:43.861665 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:43.861634    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-whv2s_0ad705ff-e172-497a-a972-108f30e7be6b/dns-node-resolver/0.log"
Jul 30 00:14:44.456873 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:44.456835    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8tftg_bb349f88-27d7-4900-9d7c-d0e7916098ab/node-ca/0.log"
Jul 30 00:14:46.288849 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:46.288822    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hlgwc_81d4f0dc-2d3a-4523-83d4-de99adcf5e03/serve-healthcheck-canary/0.log"
Jul 30 00:14:46.947719 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:46.947690    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rr5g7_2a191a01-962a-4268-93fc-639ea776d1b6/kube-rbac-proxy/0.log"
Jul 30 00:14:46.968998 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:46.968978    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rr5g7_2a191a01-962a-4268-93fc-639ea776d1b6/exporter/0.log"
Jul 30 00:14:46.993893 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:46.993872    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rr5g7_2a191a01-962a-4268-93fc-639ea776d1b6/extractor/0.log"
Jul 30 00:14:48.715545 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:48.715519    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-vsx47"
Jul 30 00:14:49.929339 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:49.929307    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-78dcd69c74-c9ztf_da621695-b3b9-4d0e-b772-b0a1c4b3a15c/manager/0.log"
Jul 30 00:14:49.956965 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:49.956934    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-dm88t_f268088a-1c18-42e9-892a-b9b0e9e860a7/openshift-lws-operator/0.log"
Jul 30 00:14:50.674582 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:50.674555    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-6t2tn_e42d3a61-ebda-44c1-9c22-deea87c1f921/server/0.log"
Jul 30 00:14:51.182846 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:51.182773    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-8588677cfb-n4kbt_5e9778d8-4555-443c-91a5-eabd88c1fccb/manager/0.log"
Jul 30 00:14:56.432770 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:56.432715    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-dftrh_66c46109-03d1-4b31-90ac-edb1642d1d04/migrator/0.log"
Jul 30 00:14:56.454737 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:56.454713    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-dftrh_66c46109-03d1-4b31-90ac-edb1642d1d04/graceful-termination/0.log"
Jul 30 00:14:56.802954 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:56.802864    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq_b6921abb-a2c9-4b8a-9cf5-6c0d694c2630/kube-storage-version-migrator-operator/1.log"
Jul 30 00:14:56.803714 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:56.803696    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq_b6921abb-a2c9-4b8a-9cf5-6c0d694c2630/kube-storage-version-migrator-operator/0.log"
Jul 30 00:14:57.749710 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:57.749683    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9cmk4_19fe18f8-824f-42c1-ac3e-e3a1bd0e1b96/kube-multus/0.log"
Jul 30 00:14:58.108760 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:58.108731    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z5plq_0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8/kube-multus-additional-cni-plugins/0.log"
Jul 30 00:14:58.131496 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:58.131478    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z5plq_0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8/egress-router-binary-copy/0.log"
Jul 30 00:14:58.153612 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:58.153596    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z5plq_0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8/cni-plugins/0.log"
Jul 30 00:14:58.175609 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:58.175589    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z5plq_0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8/bond-cni-plugin/0.log"
Jul 30 00:14:58.197922 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:58.197902    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z5plq_0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8/routeoverride-cni/0.log"
Jul 30 00:14:58.222130 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:58.222113    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z5plq_0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8/whereabouts-cni-bincopy/0.log"
Jul 30 00:14:58.244459 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:58.244442    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z5plq_0e2daf9f-e88e-4f93-a21a-f3a7bcc6a6b8/whereabouts-cni/0.log"
Jul 30 00:14:58.336282 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:58.336259    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2ksj4_cf450310-9f32-4f65-96ea-b4d676ff2df7/network-metrics-daemon/0.log"
Jul 30 00:14:58.356636 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:58.356611    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2ksj4_cf450310-9f32-4f65-96ea-b4d676ff2df7/kube-rbac-proxy/0.log"
Jul 30 00:14:59.538938 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:59.538907    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzpfl_6166a6a8-c059-4854-a4c7-51904b3265d6/ovn-controller/0.log"
Jul 30 00:14:59.558383 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:59.558343    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzpfl_6166a6a8-c059-4854-a4c7-51904b3265d6/ovn-acl-logging/0.log"
Jul 30 00:14:59.583881 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:59.583852    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzpfl_6166a6a8-c059-4854-a4c7-51904b3265d6/kube-rbac-proxy-node/0.log"
Jul 30 00:14:59.606245 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:59.606220    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzpfl_6166a6a8-c059-4854-a4c7-51904b3265d6/kube-rbac-proxy-ovn-metrics/0.log"
Jul 30 00:14:59.623491 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:59.623474    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzpfl_6166a6a8-c059-4854-a4c7-51904b3265d6/northd/0.log"
Jul 30 00:14:59.643826 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:59.643789    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzpfl_6166a6a8-c059-4854-a4c7-51904b3265d6/nbdb/0.log"
Jul 30 00:14:59.664902 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:59.664882    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzpfl_6166a6a8-c059-4854-a4c7-51904b3265d6/sbdb/0.log"
Jul 30 00:14:59.719563 ip-10-0-128-242 kubenswrapper[2581]: I0730 00:14:59.719538    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzpfl_6166a6a8-c059-4854-a4c7-51904b3265d6/ovnkube-controller/0.log"