Jul 10 18:39:36.398412 ip-10-0-135-91 systemd[1]: Starting Kubernetes Kubelet...
Jul 10 18:39:37.006697 ip-10-0-135-91 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:39:37.006697 ip-10-0-135-91 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 10 18:39:37.006697 ip-10-0-135-91 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:39:37.006697 ip-10-0-135-91 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 10 18:39:37.006697 ip-10-0-135-91 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:39:37.006697 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.856142    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 10 18:39:37.006697 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862169    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:39:37.006697 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862185    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:39:37.006697 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862188    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:39:37.006697 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862191    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:39:37.006697 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862195    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:39:37.006697 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862198    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:39:37.006697 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862201    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:39:37.006697 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862204    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:39:36.934415 ip-10-0-135-91 systemd[1]: Started Kubernetes Kubelet.
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862208    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862210    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862213    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862215    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862218    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862220    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862223    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862225    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862228    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862231    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862233    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862236    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862240    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862243    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862246    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862248    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862251    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862253    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862256    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:39:37.010299 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862259    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862261    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862264    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862266    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862269    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862271    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862274    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862276    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862279    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862281    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862284    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862286    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862289    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862291    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862294    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862297    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862299    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862303    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862305    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862308    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:39:37.174597 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862311    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862314    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862316    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862319    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862321    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862324    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862327    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862329    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862332    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862335    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862337    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862340    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862342    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862345    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862349    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862354    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862357    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862360    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862363    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862366    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:39:37.177386 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862368    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862371    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862373    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862376    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862380    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862383    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862386    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862390    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862392    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862395    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862398    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862400    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862403    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862406    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862408    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862412    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862415    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862419    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862421    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:39:37.179635 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862425    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862430    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862433    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862437    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862439    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862442    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862444    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862447    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862451    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862454    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862456    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862459    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.862462    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863199    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863206    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863209    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863212    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863215    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863218    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863221    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:39:37.182567 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863224    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863226    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863229    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863232    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863234    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863237    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863239    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863242    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863244    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863247    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863250    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863252    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863255    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863316    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863522    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863533    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863540    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863549    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863556    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863562    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:39:37.184787 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863569    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863574    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863581    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863588    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863593    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863598    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863603    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863613    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863617    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863621    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863625    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863630    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863635    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863640    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863644    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863649    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863653    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863658    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863661    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:39:37.188313 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863670    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863680    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863687    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863693    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863698    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863702    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863707    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863713    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863717    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863722    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863726    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863731    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863741    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863745    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863749    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863753    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863758    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863762    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863766    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863771    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:39:37.193143 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863775    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863782    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863788    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863793    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863803    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863807    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863812    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863821    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863829    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863833    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863837    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863843    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863849    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863854    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863859    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863863    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863873    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863878    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863882    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:39:37.197571 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863886    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863891    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863896    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863901    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863905    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863909    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863913    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863918    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863922    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863931    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863936    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863940    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863944    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.863950    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865099    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865111    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865114    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865119    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865122    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:39:37.201957 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865124    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865127    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865130    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865133    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865136    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865139    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865141    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865144    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865146    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865149    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865152    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865155    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865157    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865160    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865163    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865165    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865169    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865171    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865174    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:39:37.203162 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865177    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865182    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865186    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865189    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865191    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865196    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865199    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865201    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865204    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865207    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865209    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865212    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865214    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865217    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865220    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865222    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865225    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865228    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865230    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865235    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:39:37.203754 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865237    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865240    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865243    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865245    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865248    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865250    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865253    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865256    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865259    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865261    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865264    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865267    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865269    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865275    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865278    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865281    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865283    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865288    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865291    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:39:37.204594 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865295    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865298    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865301    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865304    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865307    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865310    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865313    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865316    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865318    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865322    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865324    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865327    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865329    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865332    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865334    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865336    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865339    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865341    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865344    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865346    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:39:37.205372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865349    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865351    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865354    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865356    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865359    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865361    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865365    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865369    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865371    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865374    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865376    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865379    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865382    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865386    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865390    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.865393    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865461    2561 flags.go:64] FLAG: --address="0.0.0.0"
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865469    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865477    2561 flags.go:64] FLAG: --anonymous-auth="true"
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865483    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865495    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 10 18:39:37.205994 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865499    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865504    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865508    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865511    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865514    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865518    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865521    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865525    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865528    2561 flags.go:64] FLAG: --cgroup-root=""
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865531    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865534    2561 flags.go:64] FLAG: --client-ca-file=""
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865537    2561 flags.go:64] FLAG: --cloud-provider="external"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865540    2561 flags.go:64] FLAG: --cluster-dns="[]"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865545    2561 flags.go:64] FLAG: --cluster-domain=""
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865548    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865551    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865554    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865559    2561 flags.go:64] FLAG: --container-log-max-files="5"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865564    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865567    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865570    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865574    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865577    2561 flags.go:64] FLAG: --contention-profiling="false"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865580    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 10 18:39:37.206611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865583    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865587    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865590    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865594    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865597    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865600    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865603    2561 flags.go:64] FLAG: --enable-load-reader="false"
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865607    2561 flags.go:64] FLAG: --enable-server="true"
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865610    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865615    2561 flags.go:64] FLAG: --event-burst="100"
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865618    2561 flags.go:64] FLAG: --event-qps="50"
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865621    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865624    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865627    2561 flags.go:64] FLAG: --eviction-hard=""
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865630    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865633    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865636    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865639    2561 flags.go:64] FLAG: --eviction-soft=""
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865642    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865645    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865648    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865651    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865654    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865657    2561 flags.go:64] FLAG: --fail-swap-on="true"
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865660    2561 flags.go:64] FLAG: --feature-gates=""
Jul 10 18:39:37.207440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865665    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865670    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865673    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865676    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865680    2561 flags.go:64] FLAG: --healthz-port="10248"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865683    2561 flags.go:64] FLAG: --help="false"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865686    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865689    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865692    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865695    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865698    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865702    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865705    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865707    2561 flags.go:64] FLAG: --image-service-endpoint=""
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865710    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865714    2561 flags.go:64] FLAG: --kube-api-burst="100"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865717    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865720    2561 flags.go:64] FLAG: --kube-api-qps="50"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865723    2561 flags.go:64] FLAG: --kube-reserved=""
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865726    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865729    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865732    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865734    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865737    2561 flags.go:64] FLAG: --lock-file=""
Jul 10 18:39:37.208145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865740    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865743    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865746    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865752    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865755    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865758    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865761    2561 flags.go:64] FLAG: --logging-format="text"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865764    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865769    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865773    2561 flags.go:64] FLAG: --manifest-url=""
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865776    2561 flags.go:64] FLAG: --manifest-url-header=""
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865781    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865784    2561 flags.go:64] FLAG: --max-open-files="1000000"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865788    2561 flags.go:64] FLAG: --max-pods="110"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865791    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865794    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865796    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865799    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865803    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865806    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865809    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865817    2561 flags.go:64] FLAG: --node-status-max-images="50"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865820    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865823    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 10 18:39:37.208803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865826    2561 flags.go:64] FLAG: --pod-cidr=""
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865829    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865835    2561 flags.go:64] FLAG: --pod-manifest-path=""
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865838    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865842    2561 flags.go:64] FLAG: --pods-per-core="0"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865845    2561 flags.go:64] FLAG: --port="10250"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865848    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865850    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0aafb928e93b88c8b"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865854    2561 flags.go:64] FLAG: --qos-reserved=""
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865856    2561 flags.go:64] FLAG: --read-only-port="10255"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865859    2561 flags.go:64] FLAG: --register-node="true"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865862    2561 flags.go:64] FLAG: --register-with-taints=""
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865867    2561 flags.go:64] FLAG: --registry-burst="10"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865870    2561 flags.go:64] FLAG: --registry-qps="5"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865872    2561 flags.go:64] FLAG: --reserved-cpus=""
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865875    2561 flags.go:64] FLAG: --reserved-memory=""
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865880    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865883    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865887    2561 flags.go:64] FLAG: --rotate-certificates="false"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865890    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865893    2561 flags.go:64] FLAG: --runonce="false"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865896    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865899    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865902    2561 flags.go:64] FLAG: --seccomp-default="false"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865905    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 10 18:39:37.211447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865908    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865911    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865914    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865917    2561 flags.go:64] FLAG: --storage-driver-password="root"
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865920    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865923    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865926    2561 flags.go:64] FLAG: --storage-driver-user="root"
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865929    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865932    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865935    2561 flags.go:64] FLAG: --system-cgroups=""
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865938    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865944    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865947    2561 flags.go:64] FLAG: --tls-cert-file=""
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865950    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865954    2561 flags.go:64] FLAG: --tls-min-version=""
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865957    2561 flags.go:64] FLAG: --tls-private-key-file=""
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865960    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865963    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865966    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865969    2561 flags.go:64] FLAG: --v="2"
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865973    2561 flags.go:64] FLAG: --version="false"
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865977    2561 flags.go:64] FLAG: --vmodule=""
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865981    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.865985    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866130    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:39:37.212278 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866136    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866141    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866144    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866148    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866150    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866153    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866156    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866158    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866161    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866164    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866166    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866169    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866172    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866174    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866177    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866180    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866183    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866186    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866189    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:39:37.215602 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866191    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866194    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866196    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866199    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866201    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866204    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866206    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866209    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866212    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866214    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866217    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866221    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866223    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866227    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866229    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866232    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866235    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866237    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866241    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:39:37.362118 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866245    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866248    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866251    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866254    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866257    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866260    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866263    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866265    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866268    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866270    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866272    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866276    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866278    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866281    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866283    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866285    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866288    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866290    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866293    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866295    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:39:37.362746 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866298    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866300    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866303    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866305    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866310    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866312    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866316    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866319    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866321    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866324    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866327    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866330    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866332    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866335    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866337    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866340    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866342    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866345    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866347    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866350    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:39:37.363342 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866352    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866355    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866357    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866360    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866362    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866365    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866368    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866370    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866372    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866375    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866377    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866381    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866384    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866387    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866389    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866392    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866395    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866398    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866402    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:39:37.363881 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.866405    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:39:37.364443 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.867187    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:39:37.364443 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.874757    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 10 18:39:37.364443 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.874776    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 10 18:39:37.364443 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.874795    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 10 18:39:37.364443 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.874801    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 10 18:39:37.364443 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874872    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:39:37.364443 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874877    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:39:37.364443 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874881    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:39:37.364443 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874884    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:39:37.364443 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874887    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:39:37.364443 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874890    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:39:37.364443 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874893    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:39:37.364443 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874895    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:39:37.364443 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874898    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874901    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874904    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874906    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874909    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874912    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874915    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874917    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874920    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874922    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874925    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874928    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874932    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874936    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874939    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874942    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874945    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874948    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874953    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:39:37.364830 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874956    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874959    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874963    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874965    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874968    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874971    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874973    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874976    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874978    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874981    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874983    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874986    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874988    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874991    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874993    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874996    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.874998    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875000    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875005    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875008    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:39:37.365388 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875010    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875013    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875017    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875020    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875023    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875025    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875028    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875030    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875033    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875035    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875038    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875042    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875044    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875047    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875050    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875056    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875058    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875061    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875063    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:39:37.365938 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875066    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875068    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875071    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875089    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875103    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875105    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875108    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875110    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875113    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875116    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875118    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875122    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875125    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875127    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875130    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875133    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875136    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875138    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875141    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875144    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:39:37.366524 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875147    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:39:37.367092 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875151    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:39:37.367092 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875153    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:39:37.367092 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875156    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:39:37.367092 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875159    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:39:37.367092 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875162    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:39:37.367092 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875166    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:39:37.367092 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875168    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:39:37.367092 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875171    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:39:37.367092 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875180    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:39:37.367092 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875183    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:39:37.367092 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875185    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:39:37.367092 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875188    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:39:37.367092 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.875193    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:39:37.367092 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875329    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875334    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875337    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875340    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875343    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875345    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875348    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875353    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875356    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875359    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875363    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875366    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875369    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875371    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875374    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875377    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875379    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875382    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875384    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:39:37.367595 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875387    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875389    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875392    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875395    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875401    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875404    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875407    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875410    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875412    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875416    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875418    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875421    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875424    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875426    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875429    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875431    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875433    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875436    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875438    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:39:37.368168 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875442    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875445    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875448    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875451    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875455    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875458    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875460    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875463    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875466    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875469    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875472    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875474    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875477    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875479    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875482    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875484    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875487    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875491    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875493    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875496    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:39:37.368877 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875498    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875501    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875503    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875506    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875509    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875511    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875514    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875516    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875519    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875521    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875524    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875526    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875529    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875531    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875534    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875536    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875541    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875544    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875547    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875549    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:39:37.369550 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875552    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875555    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875557    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875560    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875562    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875565    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875567    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875570    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875572    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875576    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875578    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875581    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875583    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875586    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875588    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875591    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875594    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875596    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875599    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875601    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:39:37.370207 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:36.875604    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:39:37.370745 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.875609    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:39:37.370745 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.875750    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 10 18:39:37.370745 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.878604    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 10 18:39:37.370745 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.880375    2561 server.go:1019] "Starting client certificate rotation"
Jul 10 18:39:37.370745 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.880476    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 18:39:37.370745 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.880506    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 18:39:37.370745 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.883428    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 18:39:37.370745 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.885116    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 18:39:37.370745 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.899393    2561 log.go:25] "Validated CRI v1 runtime API"
Jul 10 18:39:37.370745 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.902283    2561 log.go:25] "Validated CRI v1 image API"
Jul 10 18:39:37.370745 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.903052    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 10 18:39:37.370745 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.905801    2561 fs.go:135] Filesystem UUIDs: map[555fbdc1-44fd-4bc4-8456-bd467cba205e:/dev/nvme0n1p4 68338a24-9eb4-43c2-bdbf-0f8a4bb4035f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 10 18:39:37.371298 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.905817    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 10 18:39:37.371588 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.911186    2561 manager.go:216] Machine: {Timestamp:2026-07-10 18:39:36.909734626 +0000 UTC m=+0.383827502 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099596 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20f63db62a0ea3832e0d6d821191ba SystemUUID:ec20f63d-b62a-0ea3-832e-0d6d821191ba BootID:5874f230-f378-4058-b5df-ebccd24e5622 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f4:0e:b8:4d:ad Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f4:0e:b8:4d:ad Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:10:75:d5:7d:9f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 10 18:39:37.371588 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.911296    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 10 18:39:37.371588 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.911393    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 10 18:39:37.371588 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.915265    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 10 18:39:37.371588 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.915297    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-91.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 10 18:39:37.371588 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.916218    2561 topology_manager.go:138] "Creating topology manager with none policy"
Jul 10 18:39:37.371588 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.916229    2561 container_manager_linux.go:309] "Creating device plugin manager"
Jul 10 18:39:37.371588 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.916237    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.916257    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.916734    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.917546    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.922663    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.922830    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.922860    2561 kubelet.go:486] "Attempting to sync node with API server"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.922876    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.922906    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.922923    2561 kubelet.go:398] "Adding apiserver pod source"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.922935    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.923449    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.923465    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.926204    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.928303    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.928331    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.929180    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.929206    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.929214    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.929219    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.929228    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.929233    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.929241    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.929246    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.929253    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.929269    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.929281    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.929288    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.930059    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.930067    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:36.932787    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-91.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.932808    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-91.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:36.932818    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.933427    2561 server.go:1268] "Started kubelet"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.933517    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.934008    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.938673    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.941273    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.941602    2561 server.go:310] "Adding debug handlers to kubelet server"
Jul 10 18:39:37.372031 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:36.943860    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.947945    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.947954    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:36.948026    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.948703    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.948719    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.948702    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.948775    2561 factory.go:55] Registering systemd factory
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.948791    2561 factory.go:223] Registration of the systemd container factory successfully
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.948850    2561 reconstruct.go:97] "Volume reconstruction finished"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.948859    2561 reconciler.go:29] "Reconciler: start to sync state"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.948991    2561 factory.go:153] Registering CRI-O factory
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.949006    2561 factory.go:223] Registration of the crio container factory successfully
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.949053    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.949089    2561 factory.go:103] Registering Raw factory
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.949107    2561 manager.go:1196] Started watching for new ooms in manager
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.949601    2561 manager.go:349] Starting recovery of all containers
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:36.949198    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-91.ec2.internal.18c101ae9fdc9f7c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-91.ec2.internal,UID:ip-10-0-135-91.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-91.ec2.internal,},FirstTimestamp:2026-07-10 18:39:36.933400444 +0000 UTC m=+0.407493320,LastTimestamp:2026-07-10 18:39:36.933400444 +0000 UTC m=+0.407493320,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-91.ec2.internal,}"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:36.957009    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-91.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:36.957301    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.963175    2561 manager.go:354] Recovery completed
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.965997    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4vc7g"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.966972    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.968807    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.968833    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.968844    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.969359    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.969364    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.969367    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.969469    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.970089    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.970113    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:37.376791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.970123    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:36.971404    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-91.ec2.internal.18c101aea1f90e09  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-91.ec2.internal,UID:ip-10-0-135-91.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-91.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-91.ec2.internal,},FirstTimestamp:2026-07-10 18:39:36.968818185 +0000 UTC m=+0.442911060,LastTimestamp:2026-07-10 18:39:36.968818185 +0000 UTC m=+0.442911060,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-91.ec2.internal,}"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.972098    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4vc7g"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.975421    2561 policy_none.go:49] "None policy: Start"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.975447    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.975460    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:36.978193    2561 policy_none.go:47] "Start"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:37.049065    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.064914    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.066113    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.066155    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.066183    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:37.066227    2561 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.068688    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:37.149775    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:37.165589    2561 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-91.ec2.internal\" not found" node="ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:37.166718    2561 kubelet.go:2472] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.192626    2561 manager.go:337] "Starting Device Plugin manager"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:37.192698    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.192708    2561 server.go:87] "Starting device plugin registration server"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.192975    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.192986    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.193337    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.193400    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.193408    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:37.193931    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:37.193978    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.293674    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.294769    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.294798    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.294809    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.294843    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.377879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.301922    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:37.301945    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-91.ec2.internal\": node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:37.329534    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.366922    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal"]
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.367005    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.368705    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.368742    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.368758    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.371430    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.371581    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.371613    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.375765    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.375796    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.375816    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.375830    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.375799    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.375887    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.378197    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.378226    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.379020    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.379043    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.379060    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:37.409959    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-91.ec2.internal\" not found" node="ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:37.414821    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-91.ec2.internal\" not found" node="ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:37.430030    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.451643    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03ad8af5af7aef8e5b3f13cdcc15bb8d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal\" (UID: \"03ad8af5af7aef8e5b3f13cdcc15bb8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.451670    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03ad8af5af7aef8e5b3f13cdcc15bb8d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal\" (UID: \"03ad8af5af7aef8e5b3f13cdcc15bb8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.525709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.451709    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f9df3b169ce66d03811239efffa50458-config\") pod \"kube-apiserver-proxy-ip-10-0-135-91.ec2.internal\" (UID: \"f9df3b169ce66d03811239efffa50458\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.661627 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:37.530797    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 10 18:39:37.661627 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.552145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03ad8af5af7aef8e5b3f13cdcc15bb8d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal\" (UID: \"03ad8af5af7aef8e5b3f13cdcc15bb8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.661627 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.552196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03ad8af5af7aef8e5b3f13cdcc15bb8d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal\" (UID: \"03ad8af5af7aef8e5b3f13cdcc15bb8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.661627 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.552216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f9df3b169ce66d03811239efffa50458-config\") pod \"kube-apiserver-proxy-ip-10-0-135-91.ec2.internal\" (UID: \"f9df3b169ce66d03811239efffa50458\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.661627 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.552256    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03ad8af5af7aef8e5b3f13cdcc15bb8d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal\" (UID: \"03ad8af5af7aef8e5b3f13cdcc15bb8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.661627 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.552269    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03ad8af5af7aef8e5b3f13cdcc15bb8d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal\" (UID: \"03ad8af5af7aef8e5b3f13cdcc15bb8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.661627 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.552310    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f9df3b169ce66d03811239efffa50458-config\") pod \"kube-apiserver-proxy-ip-10-0-135-91.ec2.internal\" (UID: \"f9df3b169ce66d03811239efffa50458\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.661627 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:37.632883    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 10 18:39:37.806654 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.714031    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.806654 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.717779    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal"
Jul 10 18:39:37.806654 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:37.734068    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 10 18:39:37.978035 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:37.835499    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 10 18:39:37.978035 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.880337    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 10 18:39:37.978035 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.880506    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 18:39:37.978035 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:37.935566    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 10 18:39:37.980489 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.980365    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 18:34:37 +0000 UTC" deadline="2028-01-23 04:23:51.830971614 +0000 UTC"
Jul 10 18:39:37.980489 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:37.980401    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13473h44m13.850575083s"
Jul 10 18:39:38.159406 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:38.036159    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 10 18:39:38.159406 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:38.137574    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 10 18:39:38.321659 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:38.238144    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 10 18:39:38.479159 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:38.338425    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 10 18:39:38.479159 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.402156    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:39:38.479159 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.448350    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"
Jul 10 18:39:38.479159 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.455117    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:39:38.479159 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.462612    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 18:39:38.479159 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.464331    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal"
Jul 10 18:39:38.479159 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.473678    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 18:39:38.574765 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.533239    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:39:38.574765 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:38.565464    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9df3b169ce66d03811239efffa50458.slice/crio-20872bf9082d7b8b1ec826d90cd8bc80115e9d0434a407b8355ac703dd756f79 WatchSource:0}: Error finding container 20872bf9082d7b8b1ec826d90cd8bc80115e9d0434a407b8355ac703dd756f79: Status 404 returned error can't find the container with id 20872bf9082d7b8b1ec826d90cd8bc80115e9d0434a407b8355ac703dd756f79
Jul 10 18:39:38.574765 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:38.565940    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ad8af5af7aef8e5b3f13cdcc15bb8d.slice/crio-5b39eeecd57131c70f8d2b5bc15e6730cdfd0f0795e1c7c7c0d104a3a5775737 WatchSource:0}: Error finding container 5b39eeecd57131c70f8d2b5bc15e6730cdfd0f0795e1c7c7c0d104a3a5775737: Status 404 returned error can't find the container with id 5b39eeecd57131c70f8d2b5bc15e6730cdfd0f0795e1c7c7c0d104a3a5775737
Jul 10 18:39:38.574765 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.571735    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.926468    2561 apiserver.go:52] "Watching apiserver"
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.926532    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.932653    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.935160    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-nz2c7","kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal","openshift-multus/multus-additional-cni-plugins-mskk8","openshift-multus/multus-qczm9","openshift-multus/network-metrics-daemon-4rggc","openshift-network-diagnostics/network-check-target-2pkwv","openshift-network-operator/iptables-alerter-vgmh6","kube-system/konnectivity-agent-h6qx2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q","openshift-cluster-node-tuning-operator/tuned-zwpf5","openshift-dns/node-resolver-nq2tz","openshift-image-registry/node-ca-2tjml","openshift-ovn-kubernetes/ovnkube-node-sjq8k"]
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.939151    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:38.939239    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pkwv" podUID="93763faf-2f30-4f4e-aa24-77b38ff22ca3"
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.941474    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2tjml"
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.943870    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.943951    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.944124    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.944377    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.944697    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5jrmj\""
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.947818    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.947976    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.948122    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.948279    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9rffz\""
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.948527    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.948633    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.949097    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.949263    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.950861    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.951348    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.952594    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:38.952646    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rggc" podUID="17b91ece-5833-48f6-95f1-7df776a892f8"
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.952946    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 10 18:39:39.012734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.953166    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.953255    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.953541    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.953713    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.953725    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.953812    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6xsv7\""
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.955025    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h6qx2"
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.957569    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k62x8\""
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.957902    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:38.957975    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nz2c7" podUID="cde9c141-cf6b-44ff-b24d-15e1afda3808"
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.959779    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-cni-bin\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.959817    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-cni-netd\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.961502    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.959844    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9168c941-ca9d-40f0-8543-f3f103f9a50f-env-overrides\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.962279    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9168c941-ca9d-40f0-8543-f3f103f9a50f-ovn-node-metrics-cert\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.962301    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/338a7e27-acaf-4107-8de7-08dfcae53a00-os-release\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.962325    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/338a7e27-acaf-4107-8de7-08dfcae53a00-cni-binary-copy\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.962349    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/338a7e27-acaf-4107-8de7-08dfcae53a00-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.962364    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 10 18:39:39.015280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.962373    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-system-cni-dir\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.016454 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.962401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-multus-cni-dir\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.016454 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.962427    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-host-run-k8s-cni-cncf-io\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.016454 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.962447    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-host-var-lib-kubelet\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.016454 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.962461    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-etc-kubernetes\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.016454 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.962478    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-run-netns\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.016454 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.962501    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9168c941-ca9d-40f0-8543-f3f103f9a50f-ovnkube-script-lib\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.016454 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.962518    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbbx9\" (UniqueName: \"kubernetes.io/projected/9168c941-ca9d-40f0-8543-f3f103f9a50f-kube-api-access-xbbx9\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.016454 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.962539    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-host-var-lib-cni-multus\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.016454 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.962555    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c22f9db6-8f10-42be-a34d-cf75c7272b60-multus-daemon-config\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.016454 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.963847    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-slash\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.016454 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.963865    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-node-log\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.016454 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.963884    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9168c941-ca9d-40f0-8543-f3f103f9a50f-ovnkube-config\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.016454 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.963952    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 10 18:39:39.016454 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.963953    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/338a7e27-acaf-4107-8de7-08dfcae53a00-cnibin\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.019816 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.963995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/338a7e27-acaf-4107-8de7-08dfcae53a00-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.019816 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.964014    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ksgkx\""
Jul 10 18:39:39.019816 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.964018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/338a7e27-acaf-4107-8de7-08dfcae53a00-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.019816 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.964079    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg6tf\" (UniqueName: \"kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf\") pod \"network-check-target-2pkwv\" (UID: \"93763faf-2f30-4f4e-aa24-77b38ff22ca3\") " pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:39:39.019816 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.965334    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f0e0679-451d-492b-a83b-8c7662d459a7-host\") pod \"node-ca-2tjml\" (UID: \"3f0e0679-451d-492b-a83b-8c7662d459a7\") " pod="openshift-image-registry/node-ca-2tjml"
Jul 10 18:39:39.019816 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.965387    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-os-release\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.019816 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.965425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c22f9db6-8f10-42be-a34d-cf75c7272b60-cni-binary-copy\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.019816 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.965458    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-host-var-lib-cni-bin\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.019816 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.965530    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg5rf\" (UniqueName: \"kubernetes.io/projected/c22f9db6-8f10-42be-a34d-cf75c7272b60-kube-api-access-fg5rf\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.019816 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.965568    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-systemd-units\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.019816 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.965617    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-run-ovn\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.019816 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.965674    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/338a7e27-acaf-4107-8de7-08dfcae53a00-system-cni-dir\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.019816 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.965696    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg4gt\" (UniqueName: \"kubernetes.io/projected/338a7e27-acaf-4107-8de7-08dfcae53a00-kube-api-access-tg4gt\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.019816 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.965737    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-kubelet\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.019816 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.965804    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vgmh6"
Jul 10 18:39:39.020973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.965812    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-host-run-multus-certs\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.020973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.965855    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-run-systemd\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.020973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.965904    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-etc-openvswitch\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.020973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.965939    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-run-openvswitch\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.020973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.965984    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-log-socket\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.020973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.966210    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-run-ovn-kubernetes\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.020973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.966273    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.020973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.966313    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-cnibin\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.020973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.966368    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-multus-socket-dir-parent\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.020973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.966527    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-host-run-netns\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.020973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.966559    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-hostroot\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.020973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.966603    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-multus-conf-dir\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.020973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.966635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f0e0679-451d-492b-a83b-8c7662d459a7-serviceca\") pod \"node-ca-2tjml\" (UID: \"3f0e0679-451d-492b-a83b-8c7662d459a7\") " pod="openshift-image-registry/node-ca-2tjml"
Jul 10 18:39:39.020973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.966662    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-var-lib-openvswitch\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.966703    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8nwl\" (UniqueName: \"kubernetes.io/projected/3f0e0679-451d-492b-a83b-8c7662d459a7-kube-api-access-l8nwl\") pod \"node-ca-2tjml\" (UID: \"3f0e0679-451d-492b-a83b-8c7662d459a7\") " pod="openshift-image-registry/node-ca-2tjml"
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.968611    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.968807    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.969374    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.970217    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-frq6l\""
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.970371    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.971812    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.971859    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.971978    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.972237    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.972360    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vwjw7\""
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.974227    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9swvg\""
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.974235    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.974348    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.974650    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nq2tz"
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.976774    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.976796    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.976910    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gfc7j\""
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.981242    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-62kk9"
Jul 10 18:39:39.023577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:38.987659    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-62kk9"
Jul 10 18:39:39.119915 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.049754    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 10 18:39:39.119915 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.068810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-run-systemd\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.119915 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.068856    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-etc-openvswitch\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.119915 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.068890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9168c941-ca9d-40f0-8543-f3f103f9a50f-ovnkube-script-lib\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.119915 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.068945    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49n7z\" (UniqueName: \"kubernetes.io/projected/d4473d8a-7b80-4430-9006-674467a52709-kube-api-access-49n7z\") pod \"node-resolver-nq2tz\" (UID: \"d4473d8a-7b80-4430-9006-674467a52709\") " pod="openshift-dns/node-resolver-nq2tz"
Jul 10 18:39:39.119915 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.068971    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cde9c141-cf6b-44ff-b24d-15e1afda3808-dbus\") pod \"global-pull-secret-syncer-nz2c7\" (UID: \"cde9c141-cf6b-44ff-b24d-15e1afda3808\") " pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:39.119915 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.068995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc6jl\" (UniqueName: \"kubernetes.io/projected/25fb13b0-3489-45af-9326-d5e650965abe-kube-api-access-vc6jl\") pod \"iptables-alerter-vgmh6\" (UID: \"25fb13b0-3489-45af-9326-d5e650965abe\") " pod="openshift-network-operator/iptables-alerter-vgmh6"
Jul 10 18:39:39.119915 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069022    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-cnibin\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.119915 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069029    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-run-systemd\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.119915 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069047    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-multus-socket-dir-parent\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.119915 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.068994    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-etc-openvswitch\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.119915 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069072    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-multus-conf-dir\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.119915 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069124    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-multus-conf-dir\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.119915 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069125    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9168c941-ca9d-40f0-8543-f3f103f9a50f-ovn-node-metrics-cert\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.119915 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069185    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/338a7e27-acaf-4107-8de7-08dfcae53a00-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.122658 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret\") pod \"global-pull-secret-syncer-nz2c7\" (UID: \"cde9c141-cf6b-44ff-b24d-15e1afda3808\") " pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:39.122658 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069241    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-var-lib-kubelet\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.122658 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069270    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8nwl\" (UniqueName: \"kubernetes.io/projected/3f0e0679-451d-492b-a83b-8c7662d459a7-kube-api-access-l8nwl\") pod \"node-ca-2tjml\" (UID: \"3f0e0679-451d-492b-a83b-8c7662d459a7\") " pod="openshift-image-registry/node-ca-2tjml"
Jul 10 18:39:39.122658 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069295    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-cni-netd\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.122658 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069318    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9168c941-ca9d-40f0-8543-f3f103f9a50f-env-overrides\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.122658 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069347    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/338a7e27-acaf-4107-8de7-08dfcae53a00-os-release\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.122658 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069374    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-etc-sysconfig\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.122658 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069385    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-multus-socket-dir-parent\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.122658 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069400    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-multus-cni-dir\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.122658 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-etc-kubernetes\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.122658 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-cnibin\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.122658 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069470    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbbx9\" (UniqueName: \"kubernetes.io/projected/9168c941-ca9d-40f0-8543-f3f103f9a50f-kube-api-access-xbbx9\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.122658 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069507    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5cec259-ca21-4520-aa79-7049d73310c2-sys-fs\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.122658 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069672    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-multus-cni-dir\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.123607 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069708    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9168c941-ca9d-40f0-8543-f3f103f9a50f-ovnkube-script-lib\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.123607 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069720    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-etc-kubernetes\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.123607 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069780    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-cni-netd\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.123607 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069828    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r74bj\" (UniqueName: \"kubernetes.io/projected/17b91ece-5833-48f6-95f1-7df776a892f8-kube-api-access-r74bj\") pod \"network-metrics-daemon-4rggc\" (UID: \"17b91ece-5833-48f6-95f1-7df776a892f8\") " pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:39.123607 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069855    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/338a7e27-acaf-4107-8de7-08dfcae53a00-os-release\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.123607 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069876    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-etc-systemd\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.123607 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069944    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-slash\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.123607 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.069973    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5cec259-ca21-4520-aa79-7049d73310c2-socket-dir\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.123607 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.070001    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4473d8a-7b80-4430-9006-674467a52709-hosts-file\") pod \"node-resolver-nq2tz\" (UID: \"d4473d8a-7b80-4430-9006-674467a52709\") " pod="openshift-dns/node-resolver-nq2tz"
Jul 10 18:39:39.123607 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.070059    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-slash\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.123607 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.070085    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/338a7e27-acaf-4107-8de7-08dfcae53a00-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.123607 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.070100    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cde9c141-cf6b-44ff-b24d-15e1afda3808-kubelet-config\") pod \"global-pull-secret-syncer-nz2c7\" (UID: \"cde9c141-cf6b-44ff-b24d-15e1afda3808\") " pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:39.123607 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.070127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-etc-sysctl-d\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.123607 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.070482    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 10 18:39:39.125933 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.070592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bc42380f-5abf-49af-aefd-7f78735244c6-etc-tuned\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.125933 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.070653    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f0e0679-451d-492b-a83b-8c7662d459a7-host\") pod \"node-ca-2tjml\" (UID: \"3f0e0679-451d-492b-a83b-8c7662d459a7\") " pod="openshift-image-registry/node-ca-2tjml"
Jul 10 18:39:39.125933 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.070684    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5cec259-ca21-4520-aa79-7049d73310c2-device-dir\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.125933 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.070737    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-etc-sysctl-conf\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.125933 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.070766    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c22f9db6-8f10-42be-a34d-cf75c7272b60-cni-binary-copy\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.125933 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.070823    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-host-var-lib-cni-bin\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.125933 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.070855    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg5rf\" (UniqueName: \"kubernetes.io/projected/c22f9db6-8f10-42be-a34d-cf75c7272b60-kube-api-access-fg5rf\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.125933 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.070884    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/338a7e27-acaf-4107-8de7-08dfcae53a00-system-cni-dir\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.125933 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.070908    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5cec259-ca21-4520-aa79-7049d73310c2-registration-dir\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.125933 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.070938    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f0e0679-451d-492b-a83b-8c7662d459a7-host\") pod \"node-ca-2tjml\" (UID: \"3f0e0679-451d-492b-a83b-8c7662d459a7\") " pod="openshift-image-registry/node-ca-2tjml"
Jul 10 18:39:39.125933 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.070940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4473d8a-7b80-4430-9006-674467a52709-tmp-dir\") pod \"node-resolver-nq2tz\" (UID: \"d4473d8a-7b80-4430-9006-674467a52709\") " pod="openshift-dns/node-resolver-nq2tz"
Jul 10 18:39:39.125933 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.070999    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-sys\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.125933 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071028    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-kubelet\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.125933 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071054    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe5d7751-950c-406a-8a69-bf619d6f98b6-konnectivity-ca\") pod \"konnectivity-agent-h6qx2\" (UID: \"fe5d7751-950c-406a-8a69-bf619d6f98b6\") " pod="kube-system/konnectivity-agent-h6qx2"
Jul 10 18:39:39.126637 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071090    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg4gt\" (UniqueName: \"kubernetes.io/projected/338a7e27-acaf-4107-8de7-08dfcae53a00-kube-api-access-tg4gt\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.126637 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071112    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/338a7e27-acaf-4107-8de7-08dfcae53a00-system-cni-dir\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.126637 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071135    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25fb13b0-3489-45af-9326-d5e650965abe-iptables-alerter-script\") pod \"iptables-alerter-vgmh6\" (UID: \"25fb13b0-3489-45af-9326-d5e650965abe\") " pod="openshift-network-operator/iptables-alerter-vgmh6"
Jul 10 18:39:39.126637 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-host-run-multus-certs\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.126637 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071188    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-run-openvswitch\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.126637 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071188    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-host-var-lib-cni-bin\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.126637 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071214    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-log-socket\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.126637 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071239    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-run-ovn-kubernetes\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.126637 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071268    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.126637 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071294    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-host-run-netns\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.126637 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071320    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-hostroot\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.126637 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071342    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f0e0679-451d-492b-a83b-8c7662d459a7-serviceca\") pod \"node-ca-2tjml\" (UID: \"3f0e0679-451d-492b-a83b-8c7662d459a7\") " pod="openshift-image-registry/node-ca-2tjml"
Jul 10 18:39:39.126637 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071365    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-var-lib-openvswitch\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.126637 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071393    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-cni-bin\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.129947 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071417    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-host\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.129947 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071443    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/338a7e27-acaf-4107-8de7-08dfcae53a00-cni-binary-copy\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.129947 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071470    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5cec259-ca21-4520-aa79-7049d73310c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.129947 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071504    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs\") pod \"network-metrics-daemon-4rggc\" (UID: \"17b91ece-5833-48f6-95f1-7df776a892f8\") " pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:39.129947 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-etc-modprobe-d\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.129947 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071532    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c22f9db6-8f10-42be-a34d-cf75c7272b60-cni-binary-copy\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.129947 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071552    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-lib-modules\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.129947 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071598    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5cec259-ca21-4520-aa79-7049d73310c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.129947 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071625    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-system-cni-dir\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.129947 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071654    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-host-run-k8s-cni-cncf-io\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.129947 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071690    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-host-var-lib-kubelet\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.129947 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071718    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-run-netns\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.129947 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/338a7e27-acaf-4107-8de7-08dfcae53a00-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.129947 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071782    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-host-var-lib-cni-multus\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.130655 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071808    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c22f9db6-8f10-42be-a34d-cf75c7272b60-multus-daemon-config\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.130655 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071823    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-var-lib-openvswitch\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.130655 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071836    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-node-log\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.130655 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071903    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-kubelet\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.130655 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.071957    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-node-log\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.130655 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072013    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-cni-bin\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.130655 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072257    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-system-cni-dir\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.130655 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072371    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.130655 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072378    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/338a7e27-acaf-4107-8de7-08dfcae53a00-cni-binary-copy\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.130655 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072432    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-host-run-multus-certs\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.130655 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072436    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-host-run-netns\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.130655 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072461    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-host-run-k8s-cni-cncf-io\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.130655 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072481    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-hostroot\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.130655 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072515    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-run-openvswitch\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.130655 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072519    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-run-netns\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.134938 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072556    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-host-var-lib-kubelet\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.134938 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-log-socket\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.134938 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072646    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-host-run-ovn-kubernetes\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.134938 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072674    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/338a7e27-acaf-4107-8de7-08dfcae53a00-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.134938 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072781    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-host-var-lib-cni-multus\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.134938 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072884    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9168c941-ca9d-40f0-8543-f3f103f9a50f-ovnkube-config\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.134938 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072943    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/338a7e27-acaf-4107-8de7-08dfcae53a00-cnibin\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.134938 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072983    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f0e0679-451d-492b-a83b-8c7662d459a7-serviceca\") pod \"node-ca-2tjml\" (UID: \"3f0e0679-451d-492b-a83b-8c7662d459a7\") " pod="openshift-image-registry/node-ca-2tjml"
Jul 10 18:39:39.134938 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.072987    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/338a7e27-acaf-4107-8de7-08dfcae53a00-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.134938 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.073102    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9168c941-ca9d-40f0-8543-f3f103f9a50f-env-overrides\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.134938 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.073193    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/338a7e27-acaf-4107-8de7-08dfcae53a00-cnibin\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.134938 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.073234    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6tf\" (UniqueName: \"kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf\") pod \"network-check-target-2pkwv\" (UID: \"93763faf-2f30-4f4e-aa24-77b38ff22ca3\") " pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:39:39.134938 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.073284    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25fb13b0-3489-45af-9326-d5e650965abe-host-slash\") pod \"iptables-alerter-vgmh6\" (UID: \"25fb13b0-3489-45af-9326-d5e650965abe\") " pod="openshift-network-operator/iptables-alerter-vgmh6"
Jul 10 18:39:39.134938 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.073315    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9qjk\" (UniqueName: \"kubernetes.io/projected/d5cec259-ca21-4520-aa79-7049d73310c2-kube-api-access-r9qjk\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.135875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.073340    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-run\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.135875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.073364    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc42380f-5abf-49af-aefd-7f78735244c6-tmp\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.135875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.073391    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-os-release\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.135875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.073420    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-systemd-units\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.135875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.073448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-run-ovn\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.135875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.073476    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-etc-kubernetes\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.135875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.073503    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe5d7751-950c-406a-8a69-bf619d6f98b6-agent-certs\") pod \"konnectivity-agent-h6qx2\" (UID: \"fe5d7751-950c-406a-8a69-bf619d6f98b6\") " pod="kube-system/konnectivity-agent-h6qx2"
Jul 10 18:39:39.135875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.073534    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vcpj\" (UniqueName: \"kubernetes.io/projected/bc42380f-5abf-49af-aefd-7f78735244c6-kube-api-access-7vcpj\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.135875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.073627    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9168c941-ca9d-40f0-8543-f3f103f9a50f-ovnkube-config\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.135875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.073719    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c22f9db6-8f10-42be-a34d-cf75c7272b60-os-release\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.135875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.073744    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-run-ovn\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.135875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.073780    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9168c941-ca9d-40f0-8543-f3f103f9a50f-systemd-units\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.135875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.074031    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c22f9db6-8f10-42be-a34d-cf75c7272b60-multus-daemon-config\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.135875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.074103    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/338a7e27-acaf-4107-8de7-08dfcae53a00-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.135875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.074494    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal" event={"ID":"f9df3b169ce66d03811239efffa50458","Type":"ContainerStarted","Data":"20872bf9082d7b8b1ec826d90cd8bc80115e9d0434a407b8355ac703dd756f79"}
Jul 10 18:39:39.136598 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.074951    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9168c941-ca9d-40f0-8543-f3f103f9a50f-ovn-node-metrics-cert\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.136598 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.076190    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal" event={"ID":"03ad8af5af7aef8e5b3f13cdcc15bb8d","Type":"ContainerStarted","Data":"5b39eeecd57131c70f8d2b5bc15e6730cdfd0f0795e1c7c7c0d104a3a5775737"}
Jul 10 18:39:39.136598 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:39.079091    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:39:39.136598 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:39.079113    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:39:39.136598 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.079115    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8nwl\" (UniqueName: \"kubernetes.io/projected/3f0e0679-451d-492b-a83b-8c7662d459a7-kube-api-access-l8nwl\") pod \"node-ca-2tjml\" (UID: \"3f0e0679-451d-492b-a83b-8c7662d459a7\") " pod="openshift-image-registry/node-ca-2tjml"
Jul 10 18:39:39.136598 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:39.079128    2561 projected.go:196] Error preparing data for projected volume kube-api-access-xg6tf for pod openshift-network-diagnostics/network-check-target-2pkwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:39.136598 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:39.079272    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf podName:93763faf-2f30-4f4e-aa24-77b38ff22ca3 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:39.579230464 +0000 UTC m=+3.053323351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xg6tf" (UniqueName: "kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf") pod "network-check-target-2pkwv" (UID: "93763faf-2f30-4f4e-aa24-77b38ff22ca3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:39.136598 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.080495    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg4gt\" (UniqueName: \"kubernetes.io/projected/338a7e27-acaf-4107-8de7-08dfcae53a00-kube-api-access-tg4gt\") pod \"multus-additional-cni-plugins-mskk8\" (UID: \"338a7e27-acaf-4107-8de7-08dfcae53a00\") " pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.136598 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.080626    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbbx9\" (UniqueName: \"kubernetes.io/projected/9168c941-ca9d-40f0-8543-f3f103f9a50f-kube-api-access-xbbx9\") pod \"ovnkube-node-sjq8k\" (UID: \"9168c941-ca9d-40f0-8543-f3f103f9a50f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.136598 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.080687    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg5rf\" (UniqueName: \"kubernetes.io/projected/c22f9db6-8f10-42be-a34d-cf75c7272b60-kube-api-access-fg5rf\") pod \"multus-qczm9\" (UID: \"c22f9db6-8f10-42be-a34d-cf75c7272b60\") " pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.210768 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174125    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5cec259-ca21-4520-aa79-7049d73310c2-sys-fs\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.210768 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174167    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r74bj\" (UniqueName: \"kubernetes.io/projected/17b91ece-5833-48f6-95f1-7df776a892f8-kube-api-access-r74bj\") pod \"network-metrics-daemon-4rggc\" (UID: \"17b91ece-5833-48f6-95f1-7df776a892f8\") " pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:39.210768 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174192    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-etc-systemd\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.210768 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5cec259-ca21-4520-aa79-7049d73310c2-socket-dir\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.210768 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4473d8a-7b80-4430-9006-674467a52709-hosts-file\") pod \"node-resolver-nq2tz\" (UID: \"d4473d8a-7b80-4430-9006-674467a52709\") " pod="openshift-dns/node-resolver-nq2tz"
Jul 10 18:39:39.210768 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174249    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5cec259-ca21-4520-aa79-7049d73310c2-sys-fs\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.210768 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4473d8a-7b80-4430-9006-674467a52709-hosts-file\") pod \"node-resolver-nq2tz\" (UID: \"d4473d8a-7b80-4430-9006-674467a52709\") " pod="openshift-dns/node-resolver-nq2tz"
Jul 10 18:39:39.210768 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174316    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-etc-systemd\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.210768 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174409    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cde9c141-cf6b-44ff-b24d-15e1afda3808-kubelet-config\") pod \"global-pull-secret-syncer-nz2c7\" (UID: \"cde9c141-cf6b-44ff-b24d-15e1afda3808\") " pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:39.210768 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174413    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5cec259-ca21-4520-aa79-7049d73310c2-socket-dir\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.210768 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174436    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-etc-sysctl-d\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.210768 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174458    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bc42380f-5abf-49af-aefd-7f78735244c6-etc-tuned\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.210768 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174480    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5cec259-ca21-4520-aa79-7049d73310c2-device-dir\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.210768 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174499    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cde9c141-cf6b-44ff-b24d-15e1afda3808-kubelet-config\") pod \"global-pull-secret-syncer-nz2c7\" (UID: \"cde9c141-cf6b-44ff-b24d-15e1afda3808\") " pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:39.214638 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-etc-sysctl-conf\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.214638 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5cec259-ca21-4520-aa79-7049d73310c2-registration-dir\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.214638 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4473d8a-7b80-4430-9006-674467a52709-tmp-dir\") pod \"node-resolver-nq2tz\" (UID: \"d4473d8a-7b80-4430-9006-674467a52709\") " pod="openshift-dns/node-resolver-nq2tz"
Jul 10 18:39:39.214638 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174582    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5cec259-ca21-4520-aa79-7049d73310c2-device-dir\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.214638 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174596    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-sys\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.214638 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe5d7751-950c-406a-8a69-bf619d6f98b6-konnectivity-ca\") pod \"konnectivity-agent-h6qx2\" (UID: \"fe5d7751-950c-406a-8a69-bf619d6f98b6\") " pod="kube-system/konnectivity-agent-h6qx2"
Jul 10 18:39:39.214638 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174641    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5cec259-ca21-4520-aa79-7049d73310c2-registration-dir\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.214638 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174643    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25fb13b0-3489-45af-9326-d5e650965abe-iptables-alerter-script\") pod \"iptables-alerter-vgmh6\" (UID: \"25fb13b0-3489-45af-9326-d5e650965abe\") " pod="openshift-network-operator/iptables-alerter-vgmh6"
Jul 10 18:39:39.214638 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174658    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-etc-sysctl-d\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.214638 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174702    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-host\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.214638 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174742    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5cec259-ca21-4520-aa79-7049d73310c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.214638 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174768    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs\") pod \"network-metrics-daemon-4rggc\" (UID: \"17b91ece-5833-48f6-95f1-7df776a892f8\") " pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:39.214638 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-etc-modprobe-d\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.214638 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174818    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-lib-modules\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.215584 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174820    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-sys\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.215584 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174838    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-etc-sysctl-conf\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.215584 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174841    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5cec259-ca21-4520-aa79-7049d73310c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.215584 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174887    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5cec259-ca21-4520-aa79-7049d73310c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.215584 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174921    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25fb13b0-3489-45af-9326-d5e650965abe-host-slash\") pod \"iptables-alerter-vgmh6\" (UID: \"25fb13b0-3489-45af-9326-d5e650965abe\") " pod="openshift-network-operator/iptables-alerter-vgmh6"
Jul 10 18:39:39.215584 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174934    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-host\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.215584 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174949    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9qjk\" (UniqueName: \"kubernetes.io/projected/d5cec259-ca21-4520-aa79-7049d73310c2-kube-api-access-r9qjk\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.215584 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.174998    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-etc-modprobe-d\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.215584 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175025    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5cec259-ca21-4520-aa79-7049d73310c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.215584 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:39.175032    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:39.215584 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175080    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-run\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.215584 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175127    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc42380f-5abf-49af-aefd-7f78735244c6-tmp\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.215584 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175203    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25fb13b0-3489-45af-9326-d5e650965abe-iptables-alerter-script\") pod \"iptables-alerter-vgmh6\" (UID: \"25fb13b0-3489-45af-9326-d5e650965abe\") " pod="openshift-network-operator/iptables-alerter-vgmh6"
Jul 10 18:39:39.215584 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:39.175236    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs podName:17b91ece-5833-48f6-95f1-7df776a892f8 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:39.675214928 +0000 UTC m=+3.149307798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs") pod "network-metrics-daemon-4rggc" (UID: "17b91ece-5833-48f6-95f1-7df776a892f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:39.289126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25fb13b0-3489-45af-9326-d5e650965abe-host-slash\") pod \"iptables-alerter-vgmh6\" (UID: \"25fb13b0-3489-45af-9326-d5e650965abe\") " pod="openshift-network-operator/iptables-alerter-vgmh6"
Jul 10 18:39:39.289126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-lib-modules\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.289126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175334    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4473d8a-7b80-4430-9006-674467a52709-tmp-dir\") pod \"node-resolver-nq2tz\" (UID: \"d4473d8a-7b80-4430-9006-674467a52709\") " pod="openshift-dns/node-resolver-nq2tz"
Jul 10 18:39:39.289126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175374    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-etc-kubernetes\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.289126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175394    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-run\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.289126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175440    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe5d7751-950c-406a-8a69-bf619d6f98b6-agent-certs\") pod \"konnectivity-agent-h6qx2\" (UID: \"fe5d7751-950c-406a-8a69-bf619d6f98b6\") " pod="kube-system/konnectivity-agent-h6qx2"
Jul 10 18:39:39.289126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175474    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-etc-kubernetes\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.289126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175524    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vcpj\" (UniqueName: \"kubernetes.io/projected/bc42380f-5abf-49af-aefd-7f78735244c6-kube-api-access-7vcpj\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.289126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175585    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49n7z\" (UniqueName: \"kubernetes.io/projected/d4473d8a-7b80-4430-9006-674467a52709-kube-api-access-49n7z\") pod \"node-resolver-nq2tz\" (UID: \"d4473d8a-7b80-4430-9006-674467a52709\") " pod="openshift-dns/node-resolver-nq2tz"
Jul 10 18:39:39.289126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175613    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cde9c141-cf6b-44ff-b24d-15e1afda3808-dbus\") pod \"global-pull-secret-syncer-nz2c7\" (UID: \"cde9c141-cf6b-44ff-b24d-15e1afda3808\") " pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:39.289126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175637    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc6jl\" (UniqueName: \"kubernetes.io/projected/25fb13b0-3489-45af-9326-d5e650965abe-kube-api-access-vc6jl\") pod \"iptables-alerter-vgmh6\" (UID: \"25fb13b0-3489-45af-9326-d5e650965abe\") " pod="openshift-network-operator/iptables-alerter-vgmh6"
Jul 10 18:39:39.289126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175665    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe5d7751-950c-406a-8a69-bf619d6f98b6-konnectivity-ca\") pod \"konnectivity-agent-h6qx2\" (UID: \"fe5d7751-950c-406a-8a69-bf619d6f98b6\") " pod="kube-system/konnectivity-agent-h6qx2"
Jul 10 18:39:39.289126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175673    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret\") pod \"global-pull-secret-syncer-nz2c7\" (UID: \"cde9c141-cf6b-44ff-b24d-15e1afda3808\") " pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:39.289126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175701    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-var-lib-kubelet\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.289126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-etc-sysconfig\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.295103 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175745    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cde9c141-cf6b-44ff-b24d-15e1afda3808-dbus\") pod \"global-pull-secret-syncer-nz2c7\" (UID: \"cde9c141-cf6b-44ff-b24d-15e1afda3808\") " pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:39.295103 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:39.175863    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:39.295103 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:39.175929    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret podName:cde9c141-cf6b-44ff-b24d-15e1afda3808 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:39.675910866 +0000 UTC m=+3.150003748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret") pod "global-pull-secret-syncer-nz2c7" (UID: "cde9c141-cf6b-44ff-b24d-15e1afda3808") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:39.295103 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175957    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-etc-sysconfig\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.295103 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.175988    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc42380f-5abf-49af-aefd-7f78735244c6-var-lib-kubelet\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.295103 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.177385    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bc42380f-5abf-49af-aefd-7f78735244c6-etc-tuned\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.295103 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.177655    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc42380f-5abf-49af-aefd-7f78735244c6-tmp\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.295103 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.178665    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe5d7751-950c-406a-8a69-bf619d6f98b6-agent-certs\") pod \"konnectivity-agent-h6qx2\" (UID: \"fe5d7751-950c-406a-8a69-bf619d6f98b6\") " pod="kube-system/konnectivity-agent-h6qx2"
Jul 10 18:39:39.295103 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.182966    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r74bj\" (UniqueName: \"kubernetes.io/projected/17b91ece-5833-48f6-95f1-7df776a892f8-kube-api-access-r74bj\") pod \"network-metrics-daemon-4rggc\" (UID: \"17b91ece-5833-48f6-95f1-7df776a892f8\") " pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:39.295103 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.188621    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc6jl\" (UniqueName: \"kubernetes.io/projected/25fb13b0-3489-45af-9326-d5e650965abe-kube-api-access-vc6jl\") pod \"iptables-alerter-vgmh6\" (UID: \"25fb13b0-3489-45af-9326-d5e650965abe\") " pod="openshift-network-operator/iptables-alerter-vgmh6"
Jul 10 18:39:39.295103 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.188810    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49n7z\" (UniqueName: \"kubernetes.io/projected/d4473d8a-7b80-4430-9006-674467a52709-kube-api-access-49n7z\") pod \"node-resolver-nq2tz\" (UID: \"d4473d8a-7b80-4430-9006-674467a52709\") " pod="openshift-dns/node-resolver-nq2tz"
Jul 10 18:39:39.295103 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.189642    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9qjk\" (UniqueName: \"kubernetes.io/projected/d5cec259-ca21-4520-aa79-7049d73310c2-kube-api-access-r9qjk\") pod \"aws-ebs-csi-driver-node-xn96q\" (UID: \"d5cec259-ca21-4520-aa79-7049d73310c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.295103 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.191389    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vcpj\" (UniqueName: \"kubernetes.io/projected/bc42380f-5abf-49af-aefd-7f78735244c6-kube-api-access-7vcpj\") pod \"tuned-zwpf5\" (UID: \"bc42380f-5abf-49af-aefd-7f78735244c6\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.295103 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.253686    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2tjml"
Jul 10 18:39:39.295103 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.259503    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:39:39.296502 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:39.271131    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0e0679_451d_492b_a83b_8c7662d459a7.slice/crio-83ad1e60250894a24aee9ce1850cfafbfc5a358f5d9a7aa594032452fefb9ccd WatchSource:0}: Error finding container 83ad1e60250894a24aee9ce1850cfafbfc5a358f5d9a7aa594032452fefb9ccd: Status 404 returned error can't find the container with id 83ad1e60250894a24aee9ce1850cfafbfc5a358f5d9a7aa594032452fefb9ccd
Jul 10 18:39:39.296502 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:39.271881    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9168c941_ca9d_40f0_8543_f3f103f9a50f.slice/crio-7250fb6fba62ea69fe9d2be55abb2a23e0d37dbb4dfed5dd24a88bdb59626722 WatchSource:0}: Error finding container 7250fb6fba62ea69fe9d2be55abb2a23e0d37dbb4dfed5dd24a88bdb59626722: Status 404 returned error can't find the container with id 7250fb6fba62ea69fe9d2be55abb2a23e0d37dbb4dfed5dd24a88bdb59626722
Jul 10 18:39:39.296502 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.281319    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mskk8"
Jul 10 18:39:39.296502 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.285369    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qczm9"
Jul 10 18:39:39.296502 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.292328    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h6qx2"
Jul 10 18:39:39.298762 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:39.297132    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338a7e27_acaf_4107_8de7_08dfcae53a00.slice/crio-f4b6d5d74929851c0380c232b58f0bf59563ff64169800535857f12216506c77 WatchSource:0}: Error finding container f4b6d5d74929851c0380c232b58f0bf59563ff64169800535857f12216506c77: Status 404 returned error can't find the container with id f4b6d5d74929851c0380c232b58f0bf59563ff64169800535857f12216506c77
Jul 10 18:39:39.304372 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:39.304223    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22f9db6_8f10_42be_a34d_cf75c7272b60.slice/crio-6a8016c15186e82cf07bac962028bda2f03ec7df3ce04b9b64d72a7e1ec1ca4b WatchSource:0}: Error finding container 6a8016c15186e82cf07bac962028bda2f03ec7df3ce04b9b64d72a7e1ec1ca4b: Status 404 returned error can't find the container with id 6a8016c15186e82cf07bac962028bda2f03ec7df3ce04b9b64d72a7e1ec1ca4b
Jul 10 18:39:39.308954 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:39.308877    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5d7751_950c_406a_8a69_bf619d6f98b6.slice/crio-3bdc513fb76e4b865b2621286cbde06b3ddf50d04df52d1d28a7c346495b44f1 WatchSource:0}: Error finding container 3bdc513fb76e4b865b2621286cbde06b3ddf50d04df52d1d28a7c346495b44f1: Status 404 returned error can't find the container with id 3bdc513fb76e4b865b2621286cbde06b3ddf50d04df52d1d28a7c346495b44f1
Jul 10 18:39:39.356182 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.356141    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vgmh6"
Jul 10 18:39:39.364868 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.364836    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q"
Jul 10 18:39:39.368660 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:39.368629    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25fb13b0_3489_45af_9326_d5e650965abe.slice/crio-a2643598255f05fc08e45018f7daea59953c8575a636ae2b9ff3d307f3a0a677 WatchSource:0}: Error finding container a2643598255f05fc08e45018f7daea59953c8575a636ae2b9ff3d307f3a0a677: Status 404 returned error can't find the container with id a2643598255f05fc08e45018f7daea59953c8575a636ae2b9ff3d307f3a0a677
Jul 10 18:39:39.376153 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:39.376099    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5cec259_ca21_4520_aa79_7049d73310c2.slice/crio-09fcfe4217cc67241c89c27a88cc592ec86060ea88b6b8c53b63e7eeb1ad5a4e WatchSource:0}: Error finding container 09fcfe4217cc67241c89c27a88cc592ec86060ea88b6b8c53b63e7eeb1ad5a4e: Status 404 returned error can't find the container with id 09fcfe4217cc67241c89c27a88cc592ec86060ea88b6b8c53b63e7eeb1ad5a4e
Jul 10 18:39:39.379595 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.378330    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zwpf5"
Jul 10 18:39:39.387883 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.387848    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nq2tz"
Jul 10 18:39:39.390510 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:39.390482    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc42380f_5abf_49af_aefd_7f78735244c6.slice/crio-2c2bae1825fb0041950dc4860e7e81d8b39978d11bb40aae2975343dac327568 WatchSource:0}: Error finding container 2c2bae1825fb0041950dc4860e7e81d8b39978d11bb40aae2975343dac327568: Status 404 returned error can't find the container with id 2c2bae1825fb0041950dc4860e7e81d8b39978d11bb40aae2975343dac327568
Jul 10 18:39:39.399766 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:39:39.399730    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4473d8a_7b80_4430_9006_674467a52709.slice/crio-f1e24fe716c9276f34d2ddc6ae4891fa1c4b109a41701817ddc7fe6212b524a7 WatchSource:0}: Error finding container f1e24fe716c9276f34d2ddc6ae4891fa1c4b109a41701817ddc7fe6212b524a7: Status 404 returned error can't find the container with id f1e24fe716c9276f34d2ddc6ae4891fa1c4b109a41701817ddc7fe6212b524a7
Jul 10 18:39:39.579484 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.579389    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6tf\" (UniqueName: \"kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf\") pod \"network-check-target-2pkwv\" (UID: \"93763faf-2f30-4f4e-aa24-77b38ff22ca3\") " pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:39:39.579640 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:39.579608    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:39:39.579709 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:39.579641    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:39:39.579709 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:39.579665    2561 projected.go:196] Error preparing data for projected volume kube-api-access-xg6tf for pod openshift-network-diagnostics/network-check-target-2pkwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:39.579782 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:39.579751    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf podName:93763faf-2f30-4f4e-aa24-77b38ff22ca3 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:40.579727777 +0000 UTC m=+4.053820669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xg6tf" (UniqueName: "kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf") pod "network-check-target-2pkwv" (UID: "93763faf-2f30-4f4e-aa24-77b38ff22ca3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:39.681038 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.680676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret\") pod \"global-pull-secret-syncer-nz2c7\" (UID: \"cde9c141-cf6b-44ff-b24d-15e1afda3808\") " pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:39.681038 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.680761    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs\") pod \"network-metrics-daemon-4rggc\" (UID: \"17b91ece-5833-48f6-95f1-7df776a892f8\") " pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:39.681038 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:39.680890    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:39.681038 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:39.680908    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:39.681038 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:39.680957    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret podName:cde9c141-cf6b-44ff-b24d-15e1afda3808 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:40.680938553 +0000 UTC m=+4.155031434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret") pod "global-pull-secret-syncer-nz2c7" (UID: "cde9c141-cf6b-44ff-b24d-15e1afda3808") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:39.681038 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:39.680975    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs podName:17b91ece-5833-48f6-95f1-7df776a892f8 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:40.680966286 +0000 UTC m=+4.155059152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs") pod "network-metrics-daemon-4rggc" (UID: "17b91ece-5833-48f6-95f1-7df776a892f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:39.991682 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.991593    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:34:39 +0000 UTC" deadline="2028-04-03 00:51:08.748024318 +0000 UTC"
Jul 10 18:39:39.991682 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:39.991636    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15174h11m28.756393699s"
Jul 10 18:39:40.068001 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:40.067277    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:40.068001 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:40.067431    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nz2c7" podUID="cde9c141-cf6b-44ff-b24d-15e1afda3808"
Jul 10 18:39:40.068001 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:40.067865    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:39:40.068001 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:40.067959    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pkwv" podUID="93763faf-2f30-4f4e-aa24-77b38ff22ca3"
Jul 10 18:39:40.080549 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:40.080515    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nq2tz" event={"ID":"d4473d8a-7b80-4430-9006-674467a52709","Type":"ContainerStarted","Data":"f1e24fe716c9276f34d2ddc6ae4891fa1c4b109a41701817ddc7fe6212b524a7"}
Jul 10 18:39:40.081941 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:40.081887    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q" event={"ID":"d5cec259-ca21-4520-aa79-7049d73310c2","Type":"ContainerStarted","Data":"09fcfe4217cc67241c89c27a88cc592ec86060ea88b6b8c53b63e7eeb1ad5a4e"}
Jul 10 18:39:40.083853 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:40.083820    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vgmh6" event={"ID":"25fb13b0-3489-45af-9326-d5e650965abe","Type":"ContainerStarted","Data":"a2643598255f05fc08e45018f7daea59953c8575a636ae2b9ff3d307f3a0a677"}
Jul 10 18:39:40.085415 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:40.085367    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h6qx2" event={"ID":"fe5d7751-950c-406a-8a69-bf619d6f98b6","Type":"ContainerStarted","Data":"3bdc513fb76e4b865b2621286cbde06b3ddf50d04df52d1d28a7c346495b44f1"}
Jul 10 18:39:40.087628 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:40.087603    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k" event={"ID":"9168c941-ca9d-40f0-8543-f3f103f9a50f","Type":"ContainerStarted","Data":"7250fb6fba62ea69fe9d2be55abb2a23e0d37dbb4dfed5dd24a88bdb59626722"}
Jul 10 18:39:40.089213 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:40.089131    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2tjml" event={"ID":"3f0e0679-451d-492b-a83b-8c7662d459a7","Type":"ContainerStarted","Data":"83ad1e60250894a24aee9ce1850cfafbfc5a358f5d9a7aa594032452fefb9ccd"}
Jul 10 18:39:40.091374 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:40.091346    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zwpf5" event={"ID":"bc42380f-5abf-49af-aefd-7f78735244c6","Type":"ContainerStarted","Data":"2c2bae1825fb0041950dc4860e7e81d8b39978d11bb40aae2975343dac327568"}
Jul 10 18:39:40.094342 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:40.094317    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qczm9" event={"ID":"c22f9db6-8f10-42be-a34d-cf75c7272b60","Type":"ContainerStarted","Data":"6a8016c15186e82cf07bac962028bda2f03ec7df3ce04b9b64d72a7e1ec1ca4b"}
Jul 10 18:39:40.096135 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:40.096103    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mskk8" event={"ID":"338a7e27-acaf-4107-8de7-08dfcae53a00","Type":"ContainerStarted","Data":"f4b6d5d74929851c0380c232b58f0bf59563ff64169800535857f12216506c77"}
Jul 10 18:39:40.588546 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:40.588502    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6tf\" (UniqueName: \"kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf\") pod \"network-check-target-2pkwv\" (UID: \"93763faf-2f30-4f4e-aa24-77b38ff22ca3\") " pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:39:40.589303 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:40.588714    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:39:40.589303 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:40.588739    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:39:40.589303 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:40.588752    2561 projected.go:196] Error preparing data for projected volume kube-api-access-xg6tf for pod openshift-network-diagnostics/network-check-target-2pkwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:40.589303 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:40.588813    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf podName:93763faf-2f30-4f4e-aa24-77b38ff22ca3 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:42.588792507 +0000 UTC m=+6.062885390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xg6tf" (UniqueName: "kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf") pod "network-check-target-2pkwv" (UID: "93763faf-2f30-4f4e-aa24-77b38ff22ca3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:40.689741 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:40.689445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret\") pod \"global-pull-secret-syncer-nz2c7\" (UID: \"cde9c141-cf6b-44ff-b24d-15e1afda3808\") " pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:40.689741 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:40.689526    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs\") pod \"network-metrics-daemon-4rggc\" (UID: \"17b91ece-5833-48f6-95f1-7df776a892f8\") " pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:40.689741 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:40.689610    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:40.689741 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:40.689649    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:40.689741 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:40.689683    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret podName:cde9c141-cf6b-44ff-b24d-15e1afda3808 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:42.689662072 +0000 UTC m=+6.163754940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret") pod "global-pull-secret-syncer-nz2c7" (UID: "cde9c141-cf6b-44ff-b24d-15e1afda3808") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:40.689741 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:40.689708    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs podName:17b91ece-5833-48f6-95f1-7df776a892f8 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:42.689690822 +0000 UTC m=+6.163783689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs") pod "network-metrics-daemon-4rggc" (UID: "17b91ece-5833-48f6-95f1-7df776a892f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:40.991973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:40.991849    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:34:39 +0000 UTC" deadline="2027-12-22 18:13:00.399154123 +0000 UTC"
Jul 10 18:39:40.991973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:40.991919    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12719h33m19.407240093s"
Jul 10 18:39:41.069739 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:41.069171    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:41.069739 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:41.069323    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rggc" podUID="17b91ece-5833-48f6-95f1-7df776a892f8"
Jul 10 18:39:42.066956 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:42.066872    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:39:42.067639 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:42.067043    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pkwv" podUID="93763faf-2f30-4f4e-aa24-77b38ff22ca3"
Jul 10 18:39:42.067639 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:42.067519    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:42.067639 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:42.067622    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nz2c7" podUID="cde9c141-cf6b-44ff-b24d-15e1afda3808"
Jul 10 18:39:42.607095 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:42.607051    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6tf\" (UniqueName: \"kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf\") pod \"network-check-target-2pkwv\" (UID: \"93763faf-2f30-4f4e-aa24-77b38ff22ca3\") " pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:39:42.607298 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:42.607281    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:39:42.607386 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:42.607303    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:39:42.607386 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:42.607316    2561 projected.go:196] Error preparing data for projected volume kube-api-access-xg6tf for pod openshift-network-diagnostics/network-check-target-2pkwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:42.607386 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:42.607379    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf podName:93763faf-2f30-4f4e-aa24-77b38ff22ca3 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:46.607357769 +0000 UTC m=+10.081450653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xg6tf" (UniqueName: "kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf") pod "network-check-target-2pkwv" (UID: "93763faf-2f30-4f4e-aa24-77b38ff22ca3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:42.708989 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:42.708388    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret\") pod \"global-pull-secret-syncer-nz2c7\" (UID: \"cde9c141-cf6b-44ff-b24d-15e1afda3808\") " pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:42.708989 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:42.708464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs\") pod \"network-metrics-daemon-4rggc\" (UID: \"17b91ece-5833-48f6-95f1-7df776a892f8\") " pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:42.708989 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:42.708553    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:42.708989 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:42.708619    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs podName:17b91ece-5833-48f6-95f1-7df776a892f8 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:46.708598644 +0000 UTC m=+10.182691513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs") pod "network-metrics-daemon-4rggc" (UID: "17b91ece-5833-48f6-95f1-7df776a892f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:42.708989 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:42.708553    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:42.708989 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:42.708679    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret podName:cde9c141-cf6b-44ff-b24d-15e1afda3808 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:46.708665683 +0000 UTC m=+10.182758562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret") pod "global-pull-secret-syncer-nz2c7" (UID: "cde9c141-cf6b-44ff-b24d-15e1afda3808") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:43.067631 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:43.067546    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:43.068088 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:43.067718    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rggc" podUID="17b91ece-5833-48f6-95f1-7df776a892f8"
Jul 10 18:39:44.067337 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:44.066643    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:44.067337 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:44.066791    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nz2c7" podUID="cde9c141-cf6b-44ff-b24d-15e1afda3808"
Jul 10 18:39:44.067337 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:44.066644    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:39:44.067337 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:44.067295    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pkwv" podUID="93763faf-2f30-4f4e-aa24-77b38ff22ca3"
Jul 10 18:39:45.070789 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:45.069891    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:45.070789 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:45.070039    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rggc" podUID="17b91ece-5833-48f6-95f1-7df776a892f8"
Jul 10 18:39:46.067978 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:46.067424    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:46.067978 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:46.067583    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nz2c7" podUID="cde9c141-cf6b-44ff-b24d-15e1afda3808"
Jul 10 18:39:46.067978 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:46.067632    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:39:46.067978 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:46.067765    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pkwv" podUID="93763faf-2f30-4f4e-aa24-77b38ff22ca3"
Jul 10 18:39:46.644401 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:46.643692    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6tf\" (UniqueName: \"kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf\") pod \"network-check-target-2pkwv\" (UID: \"93763faf-2f30-4f4e-aa24-77b38ff22ca3\") " pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:39:46.644401 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:46.643910    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:39:46.644401 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:46.643934    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:39:46.644401 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:46.643947    2561 projected.go:196] Error preparing data for projected volume kube-api-access-xg6tf for pod openshift-network-diagnostics/network-check-target-2pkwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:46.644401 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:46.644010    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf podName:93763faf-2f30-4f4e-aa24-77b38ff22ca3 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:54.643992014 +0000 UTC m=+18.118084898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xg6tf" (UniqueName: "kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf") pod "network-check-target-2pkwv" (UID: "93763faf-2f30-4f4e-aa24-77b38ff22ca3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:46.745003 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:46.744964    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs\") pod \"network-metrics-daemon-4rggc\" (UID: \"17b91ece-5833-48f6-95f1-7df776a892f8\") " pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:46.745209 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:46.745158    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret\") pod \"global-pull-secret-syncer-nz2c7\" (UID: \"cde9c141-cf6b-44ff-b24d-15e1afda3808\") " pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:46.745209 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:46.745179    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:46.745339 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:46.745265    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs podName:17b91ece-5833-48f6-95f1-7df776a892f8 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:54.745241536 +0000 UTC m=+18.219334423 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs") pod "network-metrics-daemon-4rggc" (UID: "17b91ece-5833-48f6-95f1-7df776a892f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:46.745339 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:46.745286    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:46.745465 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:46.745349    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret podName:cde9c141-cf6b-44ff-b24d-15e1afda3808 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:54.745331753 +0000 UTC m=+18.219424637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret") pod "global-pull-secret-syncer-nz2c7" (UID: "cde9c141-cf6b-44ff-b24d-15e1afda3808") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:47.068222 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:47.068123    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:47.068383 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:47.068252    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rggc" podUID="17b91ece-5833-48f6-95f1-7df776a892f8"
Jul 10 18:39:48.067719 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:48.067619    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:39:48.068062 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:48.067763    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pkwv" podUID="93763faf-2f30-4f4e-aa24-77b38ff22ca3"
Jul 10 18:39:48.068062 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:48.067816    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:48.068062 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:48.067942    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nz2c7" podUID="cde9c141-cf6b-44ff-b24d-15e1afda3808"
Jul 10 18:39:49.067559 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:49.067478    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:49.067786 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:49.067642    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rggc" podUID="17b91ece-5833-48f6-95f1-7df776a892f8"
Jul 10 18:39:50.066850 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:50.066818    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:39:50.067034 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:50.066814    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:50.067034 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:50.066965    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pkwv" podUID="93763faf-2f30-4f4e-aa24-77b38ff22ca3"
Jul 10 18:39:50.067034 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:50.066994    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nz2c7" podUID="cde9c141-cf6b-44ff-b24d-15e1afda3808"
Jul 10 18:39:51.067661 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:51.067502    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:51.068125 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:51.067657    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rggc" podUID="17b91ece-5833-48f6-95f1-7df776a892f8"
Jul 10 18:39:52.066765 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:52.066726    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:39:52.067008 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:52.066778    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:52.067008 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:52.066885    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pkwv" podUID="93763faf-2f30-4f4e-aa24-77b38ff22ca3"
Jul 10 18:39:52.067143 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:52.067039    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nz2c7" podUID="cde9c141-cf6b-44ff-b24d-15e1afda3808"
Jul 10 18:39:53.067311 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:53.067276    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:53.067709 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:53.067424    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rggc" podUID="17b91ece-5833-48f6-95f1-7df776a892f8"
Jul 10 18:39:54.067460 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:54.067421    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:54.067941 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:54.067421    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:39:54.067941 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:54.067568    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nz2c7" podUID="cde9c141-cf6b-44ff-b24d-15e1afda3808"
Jul 10 18:39:54.067941 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:54.067657    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pkwv" podUID="93763faf-2f30-4f4e-aa24-77b38ff22ca3"
Jul 10 18:39:54.709030 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:54.708934    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6tf\" (UniqueName: \"kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf\") pod \"network-check-target-2pkwv\" (UID: \"93763faf-2f30-4f4e-aa24-77b38ff22ca3\") " pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:39:54.709205 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:54.709116    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:39:54.709205 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:54.709145    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:39:54.709205 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:54.709160    2561 projected.go:196] Error preparing data for projected volume kube-api-access-xg6tf for pod openshift-network-diagnostics/network-check-target-2pkwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:54.709374 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:54.709222    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf podName:93763faf-2f30-4f4e-aa24-77b38ff22ca3 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:10.709208333 +0000 UTC m=+34.183301196 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xg6tf" (UniqueName: "kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf") pod "network-check-target-2pkwv" (UID: "93763faf-2f30-4f4e-aa24-77b38ff22ca3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:54.809414 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:54.809373    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret\") pod \"global-pull-secret-syncer-nz2c7\" (UID: \"cde9c141-cf6b-44ff-b24d-15e1afda3808\") " pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:54.809582 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:54.809447    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs\") pod \"network-metrics-daemon-4rggc\" (UID: \"17b91ece-5833-48f6-95f1-7df776a892f8\") " pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:54.809582 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:54.809529    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:54.809656 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:54.809602    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret podName:cde9c141-cf6b-44ff-b24d-15e1afda3808 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:10.809584953 +0000 UTC m=+34.283677824 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret") pod "global-pull-secret-syncer-nz2c7" (UID: "cde9c141-cf6b-44ff-b24d-15e1afda3808") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:54.809656 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:54.809599    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:54.809656 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:54.809632    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs podName:17b91ece-5833-48f6-95f1-7df776a892f8 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:10.80962593 +0000 UTC m=+34.283718797 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs") pod "network-metrics-daemon-4rggc" (UID: "17b91ece-5833-48f6-95f1-7df776a892f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:55.066709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:55.066625    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:55.066855 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:55.066766    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rggc" podUID="17b91ece-5833-48f6-95f1-7df776a892f8"
Jul 10 18:39:56.067404 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:56.067364    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:56.067851 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:56.067363    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:39:56.067851 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:56.067514    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nz2c7" podUID="cde9c141-cf6b-44ff-b24d-15e1afda3808"
Jul 10 18:39:56.067851 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:56.067596    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pkwv" podUID="93763faf-2f30-4f4e-aa24-77b38ff22ca3"
Jul 10 18:39:57.067422 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:57.067384    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:57.067827 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:57.067484    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rggc" podUID="17b91ece-5833-48f6-95f1-7df776a892f8"
Jul 10 18:39:58.066655 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:58.066621    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:39:58.066846 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:58.066740    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pkwv" podUID="93763faf-2f30-4f4e-aa24-77b38ff22ca3"
Jul 10 18:39:58.066846 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:58.066774    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:39:58.066957 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:58.066858    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nz2c7" podUID="cde9c141-cf6b-44ff-b24d-15e1afda3808"
Jul 10 18:39:59.067090 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:39:59.066997    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:39:59.067531 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:39:59.067154    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rggc" podUID="17b91ece-5833-48f6-95f1-7df776a892f8"
Jul 10 18:40:00.066879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.066597    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:40:00.067027 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.066607    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:40:00.067027 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:00.066987    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nz2c7" podUID="cde9c141-cf6b-44ff-b24d-15e1afda3808"
Jul 10 18:40:00.067725 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:00.067024    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pkwv" podUID="93763faf-2f30-4f4e-aa24-77b38ff22ca3"
Jul 10 18:40:00.140759 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.140563    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal" event={"ID":"f9df3b169ce66d03811239efffa50458","Type":"ContainerStarted","Data":"dbb7f4c50d33eedd936c441af430bbd16b5f3b73f6b62bdba27d1e9348b57132"}
Jul 10 18:40:00.141987 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.141960    2561 generic.go:356] "Generic (PLEG): container finished" podID="03ad8af5af7aef8e5b3f13cdcc15bb8d" containerID="cb69f8b38b14d7a19f88a272320eaeade3aa2f79194ce7a889c3995346031343" exitCode=0
Jul 10 18:40:00.142152 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.142035    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal" event={"ID":"03ad8af5af7aef8e5b3f13cdcc15bb8d","Type":"ContainerDied","Data":"cb69f8b38b14d7a19f88a272320eaeade3aa2f79194ce7a889c3995346031343"}
Jul 10 18:40:00.143381 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.143354    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nq2tz" event={"ID":"d4473d8a-7b80-4430-9006-674467a52709","Type":"ContainerStarted","Data":"1a152bef398220e4f1dd26725f89020335a4263928d4b8557663b0bbc3c204ad"}
Jul 10 18:40:00.144885 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.144864    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q" event={"ID":"d5cec259-ca21-4520-aa79-7049d73310c2","Type":"ContainerStarted","Data":"3c5902e3259e3cecf7bae451c3b66f09eab3f0156cfab2650c500591a63ab213"}
Jul 10 18:40:00.146121 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.146099    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h6qx2" event={"ID":"fe5d7751-950c-406a-8a69-bf619d6f98b6","Type":"ContainerStarted","Data":"ba6eaee44d8c5c3ff02513e34671623b44d49ec4d4954adcbf9ec52ae4898753"}
Jul 10 18:40:00.150439 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.150419    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 18:40:00.150866 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.150821    2561 generic.go:356] "Generic (PLEG): container finished" podID="9168c941-ca9d-40f0-8543-f3f103f9a50f" containerID="8bc6f3f654bf7b16dc4654edd165017c0a4bb2d244ca5fe0b603d07085be5a35" exitCode=1
Jul 10 18:40:00.150955 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.150888    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k" event={"ID":"9168c941-ca9d-40f0-8543-f3f103f9a50f","Type":"ContainerStarted","Data":"5945eee62095314e25735d6d99541fac774772277331c62db9d837180d0edf09"}
Jul 10 18:40:00.151005 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.150979    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k" event={"ID":"9168c941-ca9d-40f0-8543-f3f103f9a50f","Type":"ContainerStarted","Data":"5873dd80cb2a36ec766ef4db28f78a570cfeb0b25388bc9b87f23eb5a82bd57e"}
Jul 10 18:40:00.151046 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.151032    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k" event={"ID":"9168c941-ca9d-40f0-8543-f3f103f9a50f","Type":"ContainerDied","Data":"8bc6f3f654bf7b16dc4654edd165017c0a4bb2d244ca5fe0b603d07085be5a35"}
Jul 10 18:40:00.151106 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.151096    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k" event={"ID":"9168c941-ca9d-40f0-8543-f3f103f9a50f","Type":"ContainerStarted","Data":"24391e60e41d802be226a7d648ba4e55942b1d26bf9c3f1b5c5f2f14a31158bb"}
Jul 10 18:40:00.152402 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.152380    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2tjml" event={"ID":"3f0e0679-451d-492b-a83b-8c7662d459a7","Type":"ContainerStarted","Data":"90dff940c4539f10384f869c2d443723298f11bb0685ee12c78221182e2f53b7"}
Jul 10 18:40:00.153953 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.153926    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zwpf5" event={"ID":"bc42380f-5abf-49af-aefd-7f78735244c6","Type":"ContainerStarted","Data":"964e30003c3021637974dea78ef3cc030aba9ec160eafd01ca0c0ace0233ba25"}
Jul 10 18:40:00.155363 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.155342    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qczm9" event={"ID":"c22f9db6-8f10-42be-a34d-cf75c7272b60","Type":"ContainerStarted","Data":"a31142dfc322adc3760b08b85850240c61bc02f07f45c67344d229840e544ab1"}
Jul 10 18:40:00.155440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.155402    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal" podStartSLOduration=21.155388811 podStartE2EDuration="21.155388811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:00.154691261 +0000 UTC m=+23.628784147" watchObservedRunningTime="2026-07-10 18:40:00.155388811 +0000 UTC m=+23.629481698"
Jul 10 18:40:00.156763 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.156742    2561 generic.go:356] "Generic (PLEG): container finished" podID="338a7e27-acaf-4107-8de7-08dfcae53a00" containerID="ef318276e874fefa8fb78901eae18779d38f1214d84c0d3dfd41e7d3b973f849" exitCode=0
Jul 10 18:40:00.156840 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.156775    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mskk8" event={"ID":"338a7e27-acaf-4107-8de7-08dfcae53a00","Type":"ContainerDied","Data":"ef318276e874fefa8fb78901eae18779d38f1214d84c0d3dfd41e7d3b973f849"}
Jul 10 18:40:00.184792 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.184734    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zwpf5" podStartSLOduration=2.4914276060000002 podStartE2EDuration="22.18471331s" totalImagesPullingTime="19.693285704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:00.184124962 +0000 UTC m=+23.658217869" watchObservedRunningTime="2026-07-10 18:40:00.18471331 +0000 UTC m=+23.658806195"
Jul 10 18:40:00.197371 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.197327    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2tjml" podStartSLOduration=7.044374907 podStartE2EDuration="22.197312581s" totalImagesPullingTime="15.152937674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:00.197058924 +0000 UTC m=+23.671151810" watchObservedRunningTime="2026-07-10 18:40:00.197312581 +0000 UTC m=+23.671405466"
Jul 10 18:40:00.211606 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.211564    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h6qx2" podStartSLOduration=7.0955644939999996 podStartE2EDuration="22.211547323s" totalImagesPullingTime="15.115982829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:00.211517957 +0000 UTC m=+23.685610861" watchObservedRunningTime="2026-07-10 18:40:00.211547323 +0000 UTC m=+23.685640207"
Jul 10 18:40:00.232664 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.232622    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nq2tz" podStartSLOduration=7.20807042 podStartE2EDuration="22.232602582s" totalImagesPullingTime="15.024532162s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:00.232372337 +0000 UTC m=+23.706465223" watchObservedRunningTime="2026-07-10 18:40:00.232602582 +0000 UTC m=+23.706695467"
Jul 10 18:40:00.254323 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:00.253052    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qczm9" podStartSLOduration=2.443044074 podStartE2EDuration="22.25303015s" totalImagesPullingTime="19.809986076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:00.249229111 +0000 UTC m=+23.723321997" watchObservedRunningTime="2026-07-10 18:40:00.25303015 +0000 UTC m=+23.727123037"
Jul 10 18:40:01.066925 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:01.066887    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:40:01.067133 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:01.067006    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rggc" podUID="17b91ece-5833-48f6-95f1-7df776a892f8"
Jul 10 18:40:01.161644 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:01.161609    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal" event={"ID":"03ad8af5af7aef8e5b3f13cdcc15bb8d","Type":"ContainerStarted","Data":"96db651e3c4b46a96b50e10fbe4489a3d70097c086c9f4373d530536f6d7c8c6"}
Jul 10 18:40:01.163210 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:01.163179    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vgmh6" event={"ID":"25fb13b0-3489-45af-9326-d5e650965abe","Type":"ContainerStarted","Data":"59f96039c564d46ae38e348f98a5b9c83db434dda12bb8dc051b5ac68fa8bcb1"}
Jul 10 18:40:01.166578 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:01.166560    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 18:40:01.167045 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:01.167017    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k" event={"ID":"9168c941-ca9d-40f0-8543-f3f103f9a50f","Type":"ContainerStarted","Data":"7a555dc9826acfe19b5dfcbf175e4f6e9f5ca61dfe213cc96e8825213f9f20c6"}
Jul 10 18:40:01.167163 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:01.167117    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k" event={"ID":"9168c941-ca9d-40f0-8543-f3f103f9a50f","Type":"ContainerStarted","Data":"7fdc3c947d3dbc696d018730e5444d38ffb52ee7b3cf8fbe9179be6a0d3be39d"}
Jul 10 18:40:01.177121 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:01.177058    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal" podStartSLOduration=22.177040397 podStartE2EDuration="22.177040397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:01.17700834 +0000 UTC m=+24.651101226" watchObservedRunningTime="2026-07-10 18:40:01.177040397 +0000 UTC m=+24.651133283"
Jul 10 18:40:01.190664 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:01.190621    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vgmh6" podStartSLOduration=3.7753533409999998 podStartE2EDuration="23.190607164s" totalImagesPullingTime="19.415253823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:01.190417386 +0000 UTC m=+24.664510268" watchObservedRunningTime="2026-07-10 18:40:01.190607164 +0000 UTC m=+24.664700049"
Jul 10 18:40:01.203708 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:01.203685    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h6qx2"
Jul 10 18:40:01.204287 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:01.204270    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h6qx2"
Jul 10 18:40:01.224177 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:01.224153    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 10 18:40:02.067458 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:02.067422    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:40:02.068161 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:02.067552    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nz2c7" podUID="cde9c141-cf6b-44ff-b24d-15e1afda3808"
Jul 10 18:40:02.068161 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:02.067422    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:40:02.068161 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:02.067749    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pkwv" podUID="93763faf-2f30-4f4e-aa24-77b38ff22ca3"
Jul 10 18:40:02.171053 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:02.171014    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q" event={"ID":"d5cec259-ca21-4520-aa79-7049d73310c2","Type":"ContainerStarted","Data":"6184eab5679cdb2ea30fb943e454ba58d14837cb22bc2a682198b23a40297df7"}
Jul 10 18:40:02.171667 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:02.171556    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h6qx2"
Jul 10 18:40:02.171810 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:02.171767    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h6qx2"
Jul 10 18:40:02.207555 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:02.207431    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T18:40:01.224171022Z","UUID":"53f64445-9a6c-4f96-a77e-d6a733d638b5","Handler":null,"Name":"","Endpoint":""}
Jul 10 18:40:02.209443 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:02.209420    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 10 18:40:02.209591 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:02.209466    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 10 18:40:03.067444 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:03.067405    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:40:03.067667 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:03.067553    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rggc" podUID="17b91ece-5833-48f6-95f1-7df776a892f8"
Jul 10 18:40:03.176506 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:03.176415    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q" event={"ID":"d5cec259-ca21-4520-aa79-7049d73310c2","Type":"ContainerStarted","Data":"49a5f50b5c9535d02bec9eef3638275ab90b1a2f59502ddfed727dc8b8f49632"}
Jul 10 18:40:03.180030 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:03.180003    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 18:40:03.180539 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:03.180508    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k" event={"ID":"9168c941-ca9d-40f0-8543-f3f103f9a50f","Type":"ContainerStarted","Data":"418586eb105533711ba35abdefaeb342160ac43ee8f152609fcc8604b3086a00"}
Jul 10 18:40:03.196946 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:03.196889    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xn96q" podStartSLOduration=2.626679834 podStartE2EDuration="25.196872009s" totalImagesPullingTime="22.570192175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:03.196412588 +0000 UTC m=+26.670505470" watchObservedRunningTime="2026-07-10 18:40:03.196872009 +0000 UTC m=+26.670964893"
Jul 10 18:40:04.067513 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:04.067304    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:40:04.067704 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:04.067348    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:40:04.067704 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:04.067647    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pkwv" podUID="93763faf-2f30-4f4e-aa24-77b38ff22ca3"
Jul 10 18:40:04.067704 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:04.067686    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nz2c7" podUID="cde9c141-cf6b-44ff-b24d-15e1afda3808"
Jul 10 18:40:05.066905 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:05.066814    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:40:05.067060 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:05.066973    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rggc" podUID="17b91ece-5833-48f6-95f1-7df776a892f8"
Jul 10 18:40:06.067460 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:06.067272    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:40:06.068170 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:06.067323    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:40:06.068170 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:06.067549    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nz2c7" podUID="cde9c141-cf6b-44ff-b24d-15e1afda3808"
Jul 10 18:40:06.068170 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:06.067607    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pkwv" podUID="93763faf-2f30-4f4e-aa24-77b38ff22ca3"
Jul 10 18:40:06.189309 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:06.189274    2561 generic.go:356] "Generic (PLEG): container finished" podID="338a7e27-acaf-4107-8de7-08dfcae53a00" containerID="cad2e534a08f84a90e6ff8c42aafbcead2d02f17fd76db9c92e6e52a5a1c78ab" exitCode=0
Jul 10 18:40:06.189491 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:06.189355    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mskk8" event={"ID":"338a7e27-acaf-4107-8de7-08dfcae53a00","Type":"ContainerDied","Data":"cad2e534a08f84a90e6ff8c42aafbcead2d02f17fd76db9c92e6e52a5a1c78ab"}
Jul 10 18:40:06.192645 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:06.192625    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 18:40:06.192978 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:06.192955    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k" event={"ID":"9168c941-ca9d-40f0-8543-f3f103f9a50f","Type":"ContainerStarted","Data":"e7d3c77d2b8449257f4b92f72a9ade9ac6f6b5d4731bfc2c1ab890544a9eb576"}
Jul 10 18:40:06.193266 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:06.193244    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:40:06.193493 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:06.193474    2561 scope.go:117] "RemoveContainer" containerID="8bc6f3f654bf7b16dc4654edd165017c0a4bb2d244ca5fe0b603d07085be5a35"
Jul 10 18:40:06.210982 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:06.210952    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:40:06.930278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:06.930245    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:40:07.070779 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:07.068400    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:40:07.070779 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:07.068643    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rggc" podUID="17b91ece-5833-48f6-95f1-7df776a892f8"
Jul 10 18:40:07.201447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:07.201372    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 18:40:07.202023 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:07.201983    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k" event={"ID":"9168c941-ca9d-40f0-8543-f3f103f9a50f","Type":"ContainerStarted","Data":"01751625e2053a885e2c2b69b1f9eb2b722e9f23da19616622f11bf6d4a95784"}
Jul 10 18:40:07.202313 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:07.202291    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:40:07.221312 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:07.221283    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:40:07.229854 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:07.229793    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k" podStartSLOduration=8.980165901 podStartE2EDuration="29.229773571s" totalImagesPullingTime="20.24960767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:07.228501045 +0000 UTC m=+30.702593929" watchObservedRunningTime="2026-07-10 18:40:07.229773571 +0000 UTC m=+30.703866457"
Jul 10 18:40:07.631484 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:07.631405    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2pkwv"]
Jul 10 18:40:07.631628 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:07.631552    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:40:07.631689 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:07.631667    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pkwv" podUID="93763faf-2f30-4f4e-aa24-77b38ff22ca3"
Jul 10 18:40:07.634544 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:07.634519    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4rggc"]
Jul 10 18:40:07.634660 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:07.634626    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:40:07.634739 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:07.634718    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rggc" podUID="17b91ece-5833-48f6-95f1-7df776a892f8"
Jul 10 18:40:07.637063 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:07.637041    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nz2c7"]
Jul 10 18:40:07.637165 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:07.637148    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:40:07.637261 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:07.637239    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nz2c7" podUID="cde9c141-cf6b-44ff-b24d-15e1afda3808"
Jul 10 18:40:08.208518 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:08.208485    2561 generic.go:356] "Generic (PLEG): container finished" podID="338a7e27-acaf-4107-8de7-08dfcae53a00" containerID="f04d25200daf96570506fc1a6a603523b393d0fd407c14ab2d204056cf0a6d16" exitCode=0
Jul 10 18:40:08.208928 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:08.208567    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mskk8" event={"ID":"338a7e27-acaf-4107-8de7-08dfcae53a00","Type":"ContainerDied","Data":"f04d25200daf96570506fc1a6a603523b393d0fd407c14ab2d204056cf0a6d16"}
Jul 10 18:40:09.066923 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:09.066882    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:40:09.066923 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:09.066905    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:40:09.067129 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:09.066882    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:40:09.067129 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:09.067007    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pkwv" podUID="93763faf-2f30-4f4e-aa24-77b38ff22ca3"
Jul 10 18:40:09.067221 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:09.067130    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rggc" podUID="17b91ece-5833-48f6-95f1-7df776a892f8"
Jul 10 18:40:09.067221 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:09.067207    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nz2c7" podUID="cde9c141-cf6b-44ff-b24d-15e1afda3808"
Jul 10 18:40:09.213297 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:09.213103    2561 generic.go:356] "Generic (PLEG): container finished" podID="338a7e27-acaf-4107-8de7-08dfcae53a00" containerID="a886d2cd502bf1db016e461f532a59ea2621f3fd9112d39c343005ed73e08c6d" exitCode=0
Jul 10 18:40:09.213728 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:09.213167    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mskk8" event={"ID":"338a7e27-acaf-4107-8de7-08dfcae53a00","Type":"ContainerDied","Data":"a886d2cd502bf1db016e461f532a59ea2621f3fd9112d39c343005ed73e08c6d"}
Jul 10 18:40:10.724723 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:10.724684    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6tf\" (UniqueName: \"kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf\") pod \"network-check-target-2pkwv\" (UID: \"93763faf-2f30-4f4e-aa24-77b38ff22ca3\") " pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:40:10.725037 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:10.724846    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:40:10.725037 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:10.724869    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:40:10.725037 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:10.724879    2561 projected.go:196] Error preparing data for projected volume kube-api-access-xg6tf for pod openshift-network-diagnostics/network-check-target-2pkwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:10.725037 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:10.724932    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf podName:93763faf-2f30-4f4e-aa24-77b38ff22ca3 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:42.724917685 +0000 UTC m=+66.199010561 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xg6tf" (UniqueName: "kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf") pod "network-check-target-2pkwv" (UID: "93763faf-2f30-4f4e-aa24-77b38ff22ca3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:10.825601 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:10.825557    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs\") pod \"network-metrics-daemon-4rggc\" (UID: \"17b91ece-5833-48f6-95f1-7df776a892f8\") " pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:40:10.825777 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:10.825640    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret\") pod \"global-pull-secret-syncer-nz2c7\" (UID: \"cde9c141-cf6b-44ff-b24d-15e1afda3808\") " pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:40:10.825777 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:10.825702    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:10.825777 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:10.825762    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:10.825777 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:10.825776    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs podName:17b91ece-5833-48f6-95f1-7df776a892f8 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:42.825759399 +0000 UTC m=+66.299852261 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs") pod "network-metrics-daemon-4rggc" (UID: "17b91ece-5833-48f6-95f1-7df776a892f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:10.825952 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:10.825823    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret podName:cde9c141-cf6b-44ff-b24d-15e1afda3808 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:42.825807084 +0000 UTC m=+66.299899949 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret") pod "global-pull-secret-syncer-nz2c7" (UID: "cde9c141-cf6b-44ff-b24d-15e1afda3808") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:11.067529 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:11.067418    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:40:11.067529 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:11.067467    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:40:11.067529 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:11.067423    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:40:11.067780 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:11.067554    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pkwv" podUID="93763faf-2f30-4f4e-aa24-77b38ff22ca3"
Jul 10 18:40:11.067780 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:11.067666    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rggc" podUID="17b91ece-5833-48f6-95f1-7df776a892f8"
Jul 10 18:40:11.067868 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:11.067827    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nz2c7" podUID="cde9c141-cf6b-44ff-b24d-15e1afda3808"
Jul 10 18:40:12.291093 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.291048    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeReady"
Jul 10 18:40:12.291653 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.291237    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 10 18:40:12.329979 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.329784    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d84bdf484-l56g2"]
Jul 10 18:40:12.334680 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.334652    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.337396 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.337366    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nkq5k\""
Jul 10 18:40:12.337524 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.337491    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 10 18:40:12.337722 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.337702    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 10 18:40:12.337858 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.337835    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 10 18:40:12.343684 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.343655    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 10 18:40:12.344906 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.344882    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tlhxg"]
Jul 10 18:40:12.348456 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.348437    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:40:12.349013 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.348928    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d84bdf484-l56g2"]
Jul 10 18:40:12.351189 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.350784    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 10 18:40:12.351189 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.351018    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 10 18:40:12.351189 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.351142    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4x2b4\""
Jul 10 18:40:12.351405 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.351280    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cx9r7"]
Jul 10 18:40:12.354562 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.354492    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cx9r7"
Jul 10 18:40:12.357305 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.357132    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 10 18:40:12.357305 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.357277    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 10 18:40:12.357452 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.357393    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 10 18:40:12.357504 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.357452    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g2pth\""
Jul 10 18:40:12.357903 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.357881    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tlhxg"]
Jul 10 18:40:12.370988 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.370967    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cx9r7"]
Jul 10 18:40:12.437787 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.437750    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcc5w\" (UniqueName: \"kubernetes.io/projected/91ec32f4-5c41-4dea-a301-97256b26976d-kube-api-access-lcc5w\") pod \"ingress-canary-cx9r7\" (UID: \"91ec32f4-5c41-4dea-a301-97256b26976d\") " pod="openshift-ingress-canary/ingress-canary-cx9r7"
Jul 10 18:40:12.437965 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.437794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd2816d5-3984-4084-9706-111c92941084-installation-pull-secrets\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.437965 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.437847    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:40:12.437965 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.437882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd2816d5-3984-4084-9706-111c92941084-image-registry-private-configuration\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.437965 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.437906    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/560b6122-2684-461d-a4e2-34ef3351b1bc-tmp-dir\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:40:12.438201 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.438010    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.438201 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.438043    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d4qb\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-kube-api-access-2d4qb\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.438201 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.438066    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/560b6122-2684-461d-a4e2-34ef3351b1bc-config-volume\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:40:12.438201 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.438138    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd2816d5-3984-4084-9706-111c92941084-trusted-ca\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.438201 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.438165    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh8cm\" (UniqueName: \"kubernetes.io/projected/560b6122-2684-461d-a4e2-34ef3351b1bc-kube-api-access-qh8cm\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:40:12.438437 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.438255    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd2816d5-3984-4084-9706-111c92941084-registry-certificates\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.438437 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.438295    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd2816d5-3984-4084-9706-111c92941084-ca-trust-extracted\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.438437 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.438331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-bound-sa-token\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.438437 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.438377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91ec32f4-5c41-4dea-a301-97256b26976d-cert\") pod \"ingress-canary-cx9r7\" (UID: \"91ec32f4-5c41-4dea-a301-97256b26976d\") " pod="openshift-ingress-canary/ingress-canary-cx9r7"
Jul 10 18:40:12.539707 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.539673    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh8cm\" (UniqueName: \"kubernetes.io/projected/560b6122-2684-461d-a4e2-34ef3351b1bc-kube-api-access-qh8cm\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:40:12.539707 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.539717    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd2816d5-3984-4084-9706-111c92941084-registry-certificates\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.539940 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.539740    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd2816d5-3984-4084-9706-111c92941084-ca-trust-extracted\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.539940 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.539773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-bound-sa-token\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.539940 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.539811    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91ec32f4-5c41-4dea-a301-97256b26976d-cert\") pod \"ingress-canary-cx9r7\" (UID: \"91ec32f4-5c41-4dea-a301-97256b26976d\") " pod="openshift-ingress-canary/ingress-canary-cx9r7"
Jul 10 18:40:12.539940 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.539841    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcc5w\" (UniqueName: \"kubernetes.io/projected/91ec32f4-5c41-4dea-a301-97256b26976d-kube-api-access-lcc5w\") pod \"ingress-canary-cx9r7\" (UID: \"91ec32f4-5c41-4dea-a301-97256b26976d\") " pod="openshift-ingress-canary/ingress-canary-cx9r7"
Jul 10 18:40:12.539940 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.539863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd2816d5-3984-4084-9706-111c92941084-installation-pull-secrets\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.539940 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.539897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:40:12.539940 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.539922    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd2816d5-3984-4084-9706-111c92941084-image-registry-private-configuration\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.540293 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.539947    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/560b6122-2684-461d-a4e2-34ef3351b1bc-tmp-dir\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:40:12.540293 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.539988    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.540293 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.540010    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d4qb\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-kube-api-access-2d4qb\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.540293 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.540029    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/560b6122-2684-461d-a4e2-34ef3351b1bc-config-volume\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:40:12.540293 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.540096    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd2816d5-3984-4084-9706-111c92941084-trusted-ca\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.540293 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.540153    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd2816d5-3984-4084-9706-111c92941084-ca-trust-extracted\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.540560 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:12.540372    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:40:12.540560 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.540376    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/560b6122-2684-461d-a4e2-34ef3351b1bc-tmp-dir\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:40:12.540560 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.540417    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd2816d5-3984-4084-9706-111c92941084-registry-certificates\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.540560 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:12.540390    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d84bdf484-l56g2: secret "image-registry-tls" not found
Jul 10 18:40:12.540560 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:12.540493    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls podName:bd2816d5-3984-4084-9706-111c92941084 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:13.040473947 +0000 UTC m=+36.514566820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls") pod "image-registry-d84bdf484-l56g2" (UID: "bd2816d5-3984-4084-9706-111c92941084") : secret "image-registry-tls" not found
Jul 10 18:40:12.540798 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.540709    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/560b6122-2684-461d-a4e2-34ef3351b1bc-config-volume\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:40:12.540798 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:12.540731    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:40:12.540798 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:12.540781    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls podName:560b6122-2684-461d-a4e2-34ef3351b1bc nodeName:}" failed. No retries permitted until 2026-07-10 18:40:13.040766011 +0000 UTC m=+36.514858880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls") pod "dns-default-tlhxg" (UID: "560b6122-2684-461d-a4e2-34ef3351b1bc") : secret "dns-default-metrics-tls" not found
Jul 10 18:40:12.540995 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.540976    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd2816d5-3984-4084-9706-111c92941084-trusted-ca\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.544708 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.544653    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd2816d5-3984-4084-9706-111c92941084-installation-pull-secrets\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.544708 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.544670    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91ec32f4-5c41-4dea-a301-97256b26976d-cert\") pod \"ingress-canary-cx9r7\" (UID: \"91ec32f4-5c41-4dea-a301-97256b26976d\") " pod="openshift-ingress-canary/ingress-canary-cx9r7"
Jul 10 18:40:12.544708 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.544676    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd2816d5-3984-4084-9706-111c92941084-image-registry-private-configuration\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.549159 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.549118    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d4qb\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-kube-api-access-2d4qb\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.549308 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.549284    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcc5w\" (UniqueName: \"kubernetes.io/projected/91ec32f4-5c41-4dea-a301-97256b26976d-kube-api-access-lcc5w\") pod \"ingress-canary-cx9r7\" (UID: \"91ec32f4-5c41-4dea-a301-97256b26976d\") " pod="openshift-ingress-canary/ingress-canary-cx9r7"
Jul 10 18:40:12.549973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.549950    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh8cm\" (UniqueName: \"kubernetes.io/projected/560b6122-2684-461d-a4e2-34ef3351b1bc-kube-api-access-qh8cm\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:40:12.550659 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.550161    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-bound-sa-token\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:12.668424 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:12.668390    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cx9r7"
Jul 10 18:40:13.044855 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:13.044815    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:40:13.045125 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:13.044881    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:13.045125 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:13.044980    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:40:13.045125 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:13.045004    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d84bdf484-l56g2: secret "image-registry-tls" not found
Jul 10 18:40:13.045125 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:13.045064    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls podName:bd2816d5-3984-4084-9706-111c92941084 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:14.04504127 +0000 UTC m=+37.519134150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls") pod "image-registry-d84bdf484-l56g2" (UID: "bd2816d5-3984-4084-9706-111c92941084") : secret "image-registry-tls" not found
Jul 10 18:40:13.045125 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:13.044977    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:40:13.045322 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:13.045153    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls podName:560b6122-2684-461d-a4e2-34ef3351b1bc nodeName:}" failed. No retries permitted until 2026-07-10 18:40:14.045133716 +0000 UTC m=+37.519226597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls") pod "dns-default-tlhxg" (UID: "560b6122-2684-461d-a4e2-34ef3351b1bc") : secret "dns-default-metrics-tls" not found
Jul 10 18:40:13.066843 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:13.066813    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:40:13.067010 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:13.066813    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:40:13.067010 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:13.066823    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:40:13.069571 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:13.069515    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 18:40:13.069571 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:13.069565    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 18:40:13.070438 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:13.070412    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hzzqm\""
Jul 10 18:40:13.070545 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:13.070453    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s52z2\""
Jul 10 18:40:13.070615 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:13.070596    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 18:40:13.070679 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:13.070666    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 18:40:13.764804 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:13.764768    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cx9r7"]
Jul 10 18:40:13.818126 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:40:13.818099    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ec32f4_5c41_4dea_a301_97256b26976d.slice/crio-a73b57e014ca5c99d615edb0d61c36797a249a5934b76a5524b60bfd8a14f095 WatchSource:0}: Error finding container a73b57e014ca5c99d615edb0d61c36797a249a5934b76a5524b60bfd8a14f095: Status 404 returned error can't find the container with id a73b57e014ca5c99d615edb0d61c36797a249a5934b76a5524b60bfd8a14f095
Jul 10 18:40:14.053604 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:14.053415    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:40:14.053727 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:14.053644    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:14.053727 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:14.053551    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:40:14.053804 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:14.053740    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls podName:560b6122-2684-461d-a4e2-34ef3351b1bc nodeName:}" failed. No retries permitted until 2026-07-10 18:40:16.053719838 +0000 UTC m=+39.527812721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls") pod "dns-default-tlhxg" (UID: "560b6122-2684-461d-a4e2-34ef3351b1bc") : secret "dns-default-metrics-tls" not found
Jul 10 18:40:14.053804 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:14.053775    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:40:14.053804 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:14.053786    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d84bdf484-l56g2: secret "image-registry-tls" not found
Jul 10 18:40:14.053913 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:14.053827    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls podName:bd2816d5-3984-4084-9706-111c92941084 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:16.05381612 +0000 UTC m=+39.527908983 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls") pod "image-registry-d84bdf484-l56g2" (UID: "bd2816d5-3984-4084-9706-111c92941084") : secret "image-registry-tls" not found
Jul 10 18:40:14.228791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:14.228757    2561 generic.go:356] "Generic (PLEG): container finished" podID="338a7e27-acaf-4107-8de7-08dfcae53a00" containerID="afd231efbb93bf7cb4a1eff67abc6be44993ae2c6d58c300f4cda7e64ae49f41" exitCode=0
Jul 10 18:40:14.228975 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:14.228823    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mskk8" event={"ID":"338a7e27-acaf-4107-8de7-08dfcae53a00","Type":"ContainerDied","Data":"afd231efbb93bf7cb4a1eff67abc6be44993ae2c6d58c300f4cda7e64ae49f41"}
Jul 10 18:40:14.229956 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:14.229936    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cx9r7" event={"ID":"91ec32f4-5c41-4dea-a301-97256b26976d","Type":"ContainerStarted","Data":"a73b57e014ca5c99d615edb0d61c36797a249a5934b76a5524b60bfd8a14f095"}
Jul 10 18:40:15.234970 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:15.234929    2561 generic.go:356] "Generic (PLEG): container finished" podID="338a7e27-acaf-4107-8de7-08dfcae53a00" containerID="dedd0051f321f330516514ec492784c2ddaeb0b8719816960e961c5ef1e8816a" exitCode=0
Jul 10 18:40:15.235338 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:15.234993    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mskk8" event={"ID":"338a7e27-acaf-4107-8de7-08dfcae53a00","Type":"ContainerDied","Data":"dedd0051f321f330516514ec492784c2ddaeb0b8719816960e961c5ef1e8816a"}
Jul 10 18:40:16.070414 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:16.070376    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:40:16.070595 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:16.070442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:16.070595 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:16.070541    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:40:16.070595 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:16.070556    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d84bdf484-l56g2: secret "image-registry-tls" not found
Jul 10 18:40:16.070595 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:16.070550    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:40:16.070830 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:16.070617    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls podName:bd2816d5-3984-4084-9706-111c92941084 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:20.07060204 +0000 UTC m=+43.544694924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls") pod "image-registry-d84bdf484-l56g2" (UID: "bd2816d5-3984-4084-9706-111c92941084") : secret "image-registry-tls" not found
Jul 10 18:40:16.070830 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:16.070632    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls podName:560b6122-2684-461d-a4e2-34ef3351b1bc nodeName:}" failed. No retries permitted until 2026-07-10 18:40:20.07062635 +0000 UTC m=+43.544719225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls") pod "dns-default-tlhxg" (UID: "560b6122-2684-461d-a4e2-34ef3351b1bc") : secret "dns-default-metrics-tls" not found
Jul 10 18:40:16.241627 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:16.241583    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mskk8" event={"ID":"338a7e27-acaf-4107-8de7-08dfcae53a00","Type":"ContainerStarted","Data":"5985e395b039e87d16db499fcbc77180cffdc9368b8fb27d883d3b80a86aedbd"}
Jul 10 18:40:16.264322 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:16.264260    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mskk8" podStartSLOduration=6.867946047 podStartE2EDuration="38.264237599s" totalImagesPullingTime="31.396291552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:16.263193324 +0000 UTC m=+39.737286214" watchObservedRunningTime="2026-07-10 18:40:16.264237599 +0000 UTC m=+39.738330486"
Jul 10 18:40:16.417094 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:16.417049    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:16.496440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:16.496396    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:17.353490 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:17.353406    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:17.424875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:17.424835    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:19.249553 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:19.249515    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cx9r7" event={"ID":"91ec32f4-5c41-4dea-a301-97256b26976d","Type":"ContainerStarted","Data":"712d0dbc4a26562d396ad6399f65796cd32bfe95150b71fbdcdc847ecae1d37d"}
Jul 10 18:40:19.266394 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:19.266345    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cx9r7" podStartSLOduration=2.321262795 podStartE2EDuration="7.266329443s" totalImagesPullingTime="4.945066648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:19.265586363 +0000 UTC m=+42.739679273" watchObservedRunningTime="2026-07-10 18:40:19.266329443 +0000 UTC m=+42.740422327"
Jul 10 18:40:20.104600 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:20.104560    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:40:20.104778 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:20.104622    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:20.104778 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:20.104716    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:40:20.104778 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:20.104776    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls podName:560b6122-2684-461d-a4e2-34ef3351b1bc nodeName:}" failed. No retries permitted until 2026-07-10 18:40:28.104762009 +0000 UTC m=+51.578854886 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls") pod "dns-default-tlhxg" (UID: "560b6122-2684-461d-a4e2-34ef3351b1bc") : secret "dns-default-metrics-tls" not found
Jul 10 18:40:20.104901 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:20.104724    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:40:20.104901 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:20.104814    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d84bdf484-l56g2: secret "image-registry-tls" not found
Jul 10 18:40:20.104901 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:20.104851    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls podName:bd2816d5-3984-4084-9706-111c92941084 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:28.104841159 +0000 UTC m=+51.578934026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls") pod "image-registry-d84bdf484-l56g2" (UID: "bd2816d5-3984-4084-9706-111c92941084") : secret "image-registry-tls" not found
Jul 10 18:40:22.844975 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:22.844942    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:24.709577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:24.709533    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:25.586623 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:25.586587    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:26.312903 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:26.312863    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:27.000201 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:27.000157    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:27.058547 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:27.058512    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:28.170434 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:28.170395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:28.170850 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:28.170535    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:40:28.170850 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:28.170550    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:40:28.170850 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:28.170571    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d84bdf484-l56g2: secret "image-registry-tls" not found
Jul 10 18:40:28.170850 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:28.170620    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:40:28.170850 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:28.170628    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls podName:bd2816d5-3984-4084-9706-111c92941084 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:44.170609042 +0000 UTC m=+67.644701904 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls") pod "image-registry-d84bdf484-l56g2" (UID: "bd2816d5-3984-4084-9706-111c92941084") : secret "image-registry-tls" not found
Jul 10 18:40:28.170850 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:28.170681    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls podName:560b6122-2684-461d-a4e2-34ef3351b1bc nodeName:}" failed. No retries permitted until 2026-07-10 18:40:44.170669594 +0000 UTC m=+67.644762457 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls") pod "dns-default-tlhxg" (UID: "560b6122-2684-461d-a4e2-34ef3351b1bc") : secret "dns-default-metrics-tls" not found
Jul 10 18:40:38.600641 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:38.600599    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:39.223763 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:39.223736    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sjq8k"
Jul 10 18:40:39.489619 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:39.489530    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:39.555986 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:39.555950    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:42.776389 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:42.776348    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6tf\" (UniqueName: \"kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf\") pod \"network-check-target-2pkwv\" (UID: \"93763faf-2f30-4f4e-aa24-77b38ff22ca3\") " pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:40:42.779089 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:42.779039    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 18:40:42.789302 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:42.789282    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 18:40:42.800940 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:42.800913    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg6tf\" (UniqueName: \"kubernetes.io/projected/93763faf-2f30-4f4e-aa24-77b38ff22ca3-kube-api-access-xg6tf\") pod \"network-check-target-2pkwv\" (UID: \"93763faf-2f30-4f4e-aa24-77b38ff22ca3\") " pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:40:42.877083 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:42.877035    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret\") pod \"global-pull-secret-syncer-nz2c7\" (UID: \"cde9c141-cf6b-44ff-b24d-15e1afda3808\") " pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:40:42.877254 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:42.877217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs\") pod \"network-metrics-daemon-4rggc\" (UID: \"17b91ece-5833-48f6-95f1-7df776a892f8\") " pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:40:42.879711 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:42.879693    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 18:40:42.879826 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:42.879766    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 18:40:42.887857 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:42.887837    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 10 18:40:42.887937 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:42.887900    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs podName:17b91ece-5833-48f6-95f1-7df776a892f8 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:46.887886311 +0000 UTC m=+130.361979178 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs") pod "network-metrics-daemon-4rggc" (UID: "17b91ece-5833-48f6-95f1-7df776a892f8") : secret "metrics-daemon-secret" not found
Jul 10 18:40:42.889988 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:42.889971    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cde9c141-cf6b-44ff-b24d-15e1afda3808-original-pull-secret\") pod \"global-pull-secret-syncer-nz2c7\" (UID: \"cde9c141-cf6b-44ff-b24d-15e1afda3808\") " pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:40:43.079773 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:43.079695    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nz2c7"
Jul 10 18:40:43.089139 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:43.088904    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s52z2\""
Jul 10 18:40:43.096562 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:43.096537    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:40:43.227973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:43.227940    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nz2c7"]
Jul 10 18:40:43.231405 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:40:43.231377    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde9c141_cf6b_44ff_b24d_15e1afda3808.slice/crio-cdcd3c0ea573f96147ad0a175f5e6c716e9216b881fb2ae2fe4df07d6a6989f4 WatchSource:0}: Error finding container cdcd3c0ea573f96147ad0a175f5e6c716e9216b881fb2ae2fe4df07d6a6989f4: Status 404 returned error can't find the container with id cdcd3c0ea573f96147ad0a175f5e6c716e9216b881fb2ae2fe4df07d6a6989f4
Jul 10 18:40:43.243930 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:43.243907    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2pkwv"]
Jul 10 18:40:43.246951 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:40:43.246928    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93763faf_2f30_4f4e_aa24_77b38ff22ca3.slice/crio-959a2b5559bd28e7cd0fdcafc945fb06ba4047f28dff4f1db09970f7629967c5 WatchSource:0}: Error finding container 959a2b5559bd28e7cd0fdcafc945fb06ba4047f28dff4f1db09970f7629967c5: Status 404 returned error can't find the container with id 959a2b5559bd28e7cd0fdcafc945fb06ba4047f28dff4f1db09970f7629967c5
Jul 10 18:40:43.303852 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:43.303818    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2pkwv" event={"ID":"93763faf-2f30-4f4e-aa24-77b38ff22ca3","Type":"ContainerStarted","Data":"959a2b5559bd28e7cd0fdcafc945fb06ba4047f28dff4f1db09970f7629967c5"}
Jul 10 18:40:43.304920 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:43.304895    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nz2c7" event={"ID":"cde9c141-cf6b-44ff-b24d-15e1afda3808","Type":"ContainerStarted","Data":"cdcd3c0ea573f96147ad0a175f5e6c716e9216b881fb2ae2fe4df07d6a6989f4"}
Jul 10 18:40:44.188206 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:44.188170    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:40:44.188697 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:44.188221    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:40:44.188697 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:44.188349    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:40:44.188697 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:44.188432    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls podName:560b6122-2684-461d-a4e2-34ef3351b1bc nodeName:}" failed. No retries permitted until 2026-07-10 18:41:16.188408189 +0000 UTC m=+99.662501069 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls") pod "dns-default-tlhxg" (UID: "560b6122-2684-461d-a4e2-34ef3351b1bc") : secret "dns-default-metrics-tls" not found
Jul 10 18:40:44.188697 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:44.188353    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:40:44.188697 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:44.188466    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d84bdf484-l56g2: secret "image-registry-tls" not found
Jul 10 18:40:44.188697 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:44.188548    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls podName:bd2816d5-3984-4084-9706-111c92941084 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:16.188530492 +0000 UTC m=+99.662623375 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls") pod "image-registry-d84bdf484-l56g2" (UID: "bd2816d5-3984-4084-9706-111c92941084") : secret "image-registry-tls" not found
Jul 10 18:40:45.383035 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:45.382959    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:45.451472 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:45.451426    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:45.546882 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:45.546783    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:47.321835 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:47.321798    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:47.422244 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:47.422205    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:47.481961 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:47.481925    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:48.323294 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:48.323258    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2pkwv" event={"ID":"93763faf-2f30-4f4e-aa24-77b38ff22ca3","Type":"ContainerStarted","Data":"0623a755fdf381aac1b1349bf690dd47dc93f0b5c20de51160e5934fe58b1ddb"}
Jul 10 18:40:48.323730 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:48.323353    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:40:48.324718 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:48.324697    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nz2c7" event={"ID":"cde9c141-cf6b-44ff-b24d-15e1afda3808","Type":"ContainerStarted","Data":"68797dde64b1aaaf98d2d9c766fb34bca6c05162ad680a8a28f4576c30927772"}
Jul 10 18:40:48.341381 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:48.341334    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2pkwv" podStartSLOduration=65.858940743 podStartE2EDuration="1m10.341319169s" totalImagesPullingTime="4.482378426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:48.340879972 +0000 UTC m=+71.814972857" watchObservedRunningTime="2026-07-10 18:40:48.341319169 +0000 UTC m=+71.815412054"
Jul 10 18:40:48.357559 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:48.357502    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nz2c7" podStartSLOduration=65.855147901 podStartE2EDuration="1m10.357489944s" totalImagesPullingTime="4.502342043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:48.35689475 +0000 UTC m=+71.830987637" watchObservedRunningTime="2026-07-10 18:40:48.357489944 +0000 UTC m=+71.831582828"
Jul 10 18:40:48.571271 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:48.571232    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:48.654038 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:48.653946    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:52.544624 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:52.544591    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:52.634553 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:52.634522    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:53.228694 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.227181    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2"]
Jul 10 18:40:53.232465 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.232437    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-wfk4b"]
Jul 10 18:40:53.232607 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.232571    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2"
Jul 10 18:40:53.234978 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.234949    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:40:53.235145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.234979    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-x6sjz\""
Jul 10 18:40:53.235145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.235068    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 10 18:40:53.235812 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.235798    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 10 18:40:53.235864 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.235844    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 10 18:40:53.236303 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.236289    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-wfk4b"
Jul 10 18:40:53.239045 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.239023    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 10 18:40:53.239246 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.239230    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:40:53.239314 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.239232    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-vxlrk\""
Jul 10 18:40:53.239489 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.239466    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 10 18:40:53.239881 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.239566    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 10 18:40:53.242600 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.242579    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2"]
Jul 10 18:40:53.244868 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.244844    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 10 18:40:53.248236 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.248212    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-wfk4b"]
Jul 10 18:40:53.360574 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.360537    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4txgg\" (UniqueName: \"kubernetes.io/projected/a4adda52-8024-4b86-a7f0-6f6ee4d9e731-kube-api-access-4txgg\") pod \"service-ca-operator-7695fc6c8c-5j8c2\" (UID: \"a4adda52-8024-4b86-a7f0-6f6ee4d9e731\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2"
Jul 10 18:40:53.360756 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.360583    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4adda52-8024-4b86-a7f0-6f6ee4d9e731-serving-cert\") pod \"service-ca-operator-7695fc6c8c-5j8c2\" (UID: \"a4adda52-8024-4b86-a7f0-6f6ee4d9e731\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2"
Jul 10 18:40:53.360756 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.360660    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57cz2\" (UniqueName: \"kubernetes.io/projected/8cd32a9e-c8d4-4e46-8737-e786070a64f6-kube-api-access-57cz2\") pod \"console-operator-7866497494-wfk4b\" (UID: \"8cd32a9e-c8d4-4e46-8737-e786070a64f6\") " pod="openshift-console-operator/console-operator-7866497494-wfk4b"
Jul 10 18:40:53.360756 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.360699    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8cd32a9e-c8d4-4e46-8737-e786070a64f6-config\") pod \"console-operator-7866497494-wfk4b\" (UID: \"8cd32a9e-c8d4-4e46-8737-e786070a64f6\") " pod="openshift-console-operator/console-operator-7866497494-wfk4b"
Jul 10 18:40:53.360756 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.360718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4adda52-8024-4b86-a7f0-6f6ee4d9e731-config\") pod \"service-ca-operator-7695fc6c8c-5j8c2\" (UID: \"a4adda52-8024-4b86-a7f0-6f6ee4d9e731\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2"
Jul 10 18:40:53.360756 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.360732    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cd32a9e-c8d4-4e46-8737-e786070a64f6-trusted-ca\") pod \"console-operator-7866497494-wfk4b\" (UID: \"8cd32a9e-c8d4-4e46-8737-e786070a64f6\") " pod="openshift-console-operator/console-operator-7866497494-wfk4b"
Jul 10 18:40:53.360756 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.360749    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8cd32a9e-c8d4-4e46-8737-e786070a64f6-serving-cert\") pod \"console-operator-7866497494-wfk4b\" (UID: \"8cd32a9e-c8d4-4e46-8737-e786070a64f6\") " pod="openshift-console-operator/console-operator-7866497494-wfk4b"
Jul 10 18:40:53.405910 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.405878    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:53.461308 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.461273    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4adda52-8024-4b86-a7f0-6f6ee4d9e731-config\") pod \"service-ca-operator-7695fc6c8c-5j8c2\" (UID: \"a4adda52-8024-4b86-a7f0-6f6ee4d9e731\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2"
Jul 10 18:40:53.461443 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.461316    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cd32a9e-c8d4-4e46-8737-e786070a64f6-trusted-ca\") pod \"console-operator-7866497494-wfk4b\" (UID: \"8cd32a9e-c8d4-4e46-8737-e786070a64f6\") " pod="openshift-console-operator/console-operator-7866497494-wfk4b"
Jul 10 18:40:53.461443 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.461359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8cd32a9e-c8d4-4e46-8737-e786070a64f6-serving-cert\") pod \"console-operator-7866497494-wfk4b\" (UID: \"8cd32a9e-c8d4-4e46-8737-e786070a64f6\") " pod="openshift-console-operator/console-operator-7866497494-wfk4b"
Jul 10 18:40:53.461443 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.461406    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4txgg\" (UniqueName: \"kubernetes.io/projected/a4adda52-8024-4b86-a7f0-6f6ee4d9e731-kube-api-access-4txgg\") pod \"service-ca-operator-7695fc6c8c-5j8c2\" (UID: \"a4adda52-8024-4b86-a7f0-6f6ee4d9e731\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2"
Jul 10 18:40:53.461443 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.461429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4adda52-8024-4b86-a7f0-6f6ee4d9e731-serving-cert\") pod \"service-ca-operator-7695fc6c8c-5j8c2\" (UID: \"a4adda52-8024-4b86-a7f0-6f6ee4d9e731\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2"
Jul 10 18:40:53.461630 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.461486    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57cz2\" (UniqueName: \"kubernetes.io/projected/8cd32a9e-c8d4-4e46-8737-e786070a64f6-kube-api-access-57cz2\") pod \"console-operator-7866497494-wfk4b\" (UID: \"8cd32a9e-c8d4-4e46-8737-e786070a64f6\") " pod="openshift-console-operator/console-operator-7866497494-wfk4b"
Jul 10 18:40:53.461630 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.461533    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8cd32a9e-c8d4-4e46-8737-e786070a64f6-config\") pod \"console-operator-7866497494-wfk4b\" (UID: \"8cd32a9e-c8d4-4e46-8737-e786070a64f6\") " pod="openshift-console-operator/console-operator-7866497494-wfk4b"
Jul 10 18:40:53.461884 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.461864    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4adda52-8024-4b86-a7f0-6f6ee4d9e731-config\") pod \"service-ca-operator-7695fc6c8c-5j8c2\" (UID: \"a4adda52-8024-4b86-a7f0-6f6ee4d9e731\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2"
Jul 10 18:40:53.462287 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.462261    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8cd32a9e-c8d4-4e46-8737-e786070a64f6-config\") pod \"console-operator-7866497494-wfk4b\" (UID: \"8cd32a9e-c8d4-4e46-8737-e786070a64f6\") " pod="openshift-console-operator/console-operator-7866497494-wfk4b"
Jul 10 18:40:53.462777 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.462752    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cd32a9e-c8d4-4e46-8737-e786070a64f6-trusted-ca\") pod \"console-operator-7866497494-wfk4b\" (UID: \"8cd32a9e-c8d4-4e46-8737-e786070a64f6\") " pod="openshift-console-operator/console-operator-7866497494-wfk4b"
Jul 10 18:40:53.463804 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.463783    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4adda52-8024-4b86-a7f0-6f6ee4d9e731-serving-cert\") pod \"service-ca-operator-7695fc6c8c-5j8c2\" (UID: \"a4adda52-8024-4b86-a7f0-6f6ee4d9e731\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2"
Jul 10 18:40:53.463931 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.463911    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8cd32a9e-c8d4-4e46-8737-e786070a64f6-serving-cert\") pod \"console-operator-7866497494-wfk4b\" (UID: \"8cd32a9e-c8d4-4e46-8737-e786070a64f6\") " pod="openshift-console-operator/console-operator-7866497494-wfk4b"
Jul 10 18:40:53.470796 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.470764    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4txgg\" (UniqueName: \"kubernetes.io/projected/a4adda52-8024-4b86-a7f0-6f6ee4d9e731-kube-api-access-4txgg\") pod \"service-ca-operator-7695fc6c8c-5j8c2\" (UID: \"a4adda52-8024-4b86-a7f0-6f6ee4d9e731\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2"
Jul 10 18:40:53.470884 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.470858    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57cz2\" (UniqueName: \"kubernetes.io/projected/8cd32a9e-c8d4-4e46-8737-e786070a64f6-kube-api-access-57cz2\") pod \"console-operator-7866497494-wfk4b\" (UID: \"8cd32a9e-c8d4-4e46-8737-e786070a64f6\") " pod="openshift-console-operator/console-operator-7866497494-wfk4b"
Jul 10 18:40:53.479097 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.479006    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:53.543799 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.543761    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2"
Jul 10 18:40:53.549232 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.549208    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-wfk4b"
Jul 10 18:40:53.660242 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.660211    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9"]
Jul 10 18:40:53.666014 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.665982    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9"
Jul 10 18:40:53.668602 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.668565    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qz5c7\""
Jul 10 18:40:53.668602 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.668583    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:40:53.668828 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.668808    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 10 18:40:53.668954 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.668893    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 10 18:40:53.669023 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.668962    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 10 18:40:53.674177 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.673524    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9"]
Jul 10 18:40:53.683496 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.683467    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2"]
Jul 10 18:40:53.687367 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:40:53.687346    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4adda52_8024_4b86_a7f0_6f6ee4d9e731.slice/crio-2c471bbe2053f1f313a22e0822110c1705859343dd2aa78e680cdee1f5804c0a WatchSource:0}: Error finding container 2c471bbe2053f1f313a22e0822110c1705859343dd2aa78e680cdee1f5804c0a: Status 404 returned error can't find the container with id 2c471bbe2053f1f313a22e0822110c1705859343dd2aa78e680cdee1f5804c0a
Jul 10 18:40:53.702335 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.702310    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-wfk4b"]
Jul 10 18:40:53.704954 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:40:53.704929    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd32a9e_c8d4_4e46_8737_e786070a64f6.slice/crio-7a78a33e4b5b4ea033bf71e619ddcf0c2eed8fecf851a455296c8173f5965934 WatchSource:0}: Error finding container 7a78a33e4b5b4ea033bf71e619ddcf0c2eed8fecf851a455296c8173f5965934: Status 404 returned error can't find the container with id 7a78a33e4b5b4ea033bf71e619ddcf0c2eed8fecf851a455296c8173f5965934
Jul 10 18:40:53.763385 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.763288    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27e2c236-1c19-48b0-ad1a-38d02e492b6c-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-v7qt9\" (UID: \"27e2c236-1c19-48b0-ad1a-38d02e492b6c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9"
Jul 10 18:40:53.763385 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.763337    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27e2c236-1c19-48b0-ad1a-38d02e492b6c-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-v7qt9\" (UID: \"27e2c236-1c19-48b0-ad1a-38d02e492b6c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9"
Jul 10 18:40:53.763385 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.763367    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k27ds\" (UniqueName: \"kubernetes.io/projected/27e2c236-1c19-48b0-ad1a-38d02e492b6c-kube-api-access-k27ds\") pod \"kube-storage-version-migrator-operator-799cb7f958-v7qt9\" (UID: \"27e2c236-1c19-48b0-ad1a-38d02e492b6c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9"
Jul 10 18:40:53.863917 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.863872    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27e2c236-1c19-48b0-ad1a-38d02e492b6c-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-v7qt9\" (UID: \"27e2c236-1c19-48b0-ad1a-38d02e492b6c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9"
Jul 10 18:40:53.864128 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.863925    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27e2c236-1c19-48b0-ad1a-38d02e492b6c-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-v7qt9\" (UID: \"27e2c236-1c19-48b0-ad1a-38d02e492b6c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9"
Jul 10 18:40:53.864128 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.863953    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k27ds\" (UniqueName: \"kubernetes.io/projected/27e2c236-1c19-48b0-ad1a-38d02e492b6c-kube-api-access-k27ds\") pod \"kube-storage-version-migrator-operator-799cb7f958-v7qt9\" (UID: \"27e2c236-1c19-48b0-ad1a-38d02e492b6c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9"
Jul 10 18:40:53.864576 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.864552    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27e2c236-1c19-48b0-ad1a-38d02e492b6c-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-v7qt9\" (UID: \"27e2c236-1c19-48b0-ad1a-38d02e492b6c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9"
Jul 10 18:40:53.866160 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.866143    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27e2c236-1c19-48b0-ad1a-38d02e492b6c-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-v7qt9\" (UID: \"27e2c236-1c19-48b0-ad1a-38d02e492b6c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9"
Jul 10 18:40:53.873327 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.873307    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k27ds\" (UniqueName: \"kubernetes.io/projected/27e2c236-1c19-48b0-ad1a-38d02e492b6c-kube-api-access-k27ds\") pod \"kube-storage-version-migrator-operator-799cb7f958-v7qt9\" (UID: \"27e2c236-1c19-48b0-ad1a-38d02e492b6c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9"
Jul 10 18:40:53.976975 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:53.976937    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9"
Jul 10 18:40:54.114467 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:54.114431    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9"]
Jul 10 18:40:54.118158 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:40:54.118130    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e2c236_1c19_48b0_ad1a_38d02e492b6c.slice/crio-91c06abffc11279728471c53a7d1052f851a1f0a34d08bd4fd3bdbf4cf53706f WatchSource:0}: Error finding container 91c06abffc11279728471c53a7d1052f851a1f0a34d08bd4fd3bdbf4cf53706f: Status 404 returned error can't find the container with id 91c06abffc11279728471c53a7d1052f851a1f0a34d08bd4fd3bdbf4cf53706f
Jul 10 18:40:54.342135 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:54.342028    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-wfk4b" event={"ID":"8cd32a9e-c8d4-4e46-8737-e786070a64f6","Type":"ContainerStarted","Data":"7a78a33e4b5b4ea033bf71e619ddcf0c2eed8fecf851a455296c8173f5965934"}
Jul 10 18:40:54.344695 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:54.344660    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2" event={"ID":"a4adda52-8024-4b86-a7f0-6f6ee4d9e731","Type":"ContainerStarted","Data":"2c471bbe2053f1f313a22e0822110c1705859343dd2aa78e680cdee1f5804c0a"}
Jul 10 18:40:54.346127 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:54.346095    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9" event={"ID":"27e2c236-1c19-48b0-ad1a-38d02e492b6c","Type":"ContainerStarted","Data":"91c06abffc11279728471c53a7d1052f851a1f0a34d08bd4fd3bdbf4cf53706f"}
Jul 10 18:40:55.655058 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:55.655015    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:55.718833 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:55.718776    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:56.333842 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:56.333802    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-wfc8f"]
Jul 10 18:40:56.338324 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:56.338290    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-wfc8f"
Jul 10 18:40:56.341957 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:56.341927    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-k4ccb\""
Jul 10 18:40:56.347523 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:56.347448    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-wfc8f"]
Jul 10 18:40:56.482932 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:56.482905    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjkxp\" (UniqueName: \"kubernetes.io/projected/1ac48545-82a6-41bc-87e8-890c729f1b94-kube-api-access-jjkxp\") pod \"network-check-source-869dcfcdcd-wfc8f\" (UID: \"1ac48545-82a6-41bc-87e8-890c729f1b94\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-wfc8f"
Jul 10 18:40:56.584305 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:56.584201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjkxp\" (UniqueName: \"kubernetes.io/projected/1ac48545-82a6-41bc-87e8-890c729f1b94-kube-api-access-jjkxp\") pod \"network-check-source-869dcfcdcd-wfc8f\" (UID: \"1ac48545-82a6-41bc-87e8-890c729f1b94\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-wfc8f"
Jul 10 18:40:56.594245 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:56.594214    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjkxp\" (UniqueName: \"kubernetes.io/projected/1ac48545-82a6-41bc-87e8-890c729f1b94-kube-api-access-jjkxp\") pod \"network-check-source-869dcfcdcd-wfc8f\" (UID: \"1ac48545-82a6-41bc-87e8-890c729f1b94\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-wfc8f"
Jul 10 18:40:56.650053 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:56.650011    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-wfc8f"
Jul 10 18:40:56.803211 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:56.803173    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-wfc8f"]
Jul 10 18:40:56.808284 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:40:56.808253    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac48545_82a6_41bc_87e8_890c729f1b94.slice/crio-01810e7cfdf88076a633ea7100088198ecc841d1f7b32ef82a8b8ece568cf191 WatchSource:0}: Error finding container 01810e7cfdf88076a633ea7100088198ecc841d1f7b32ef82a8b8ece568cf191: Status 404 returned error can't find the container with id 01810e7cfdf88076a633ea7100088198ecc841d1f7b32ef82a8b8ece568cf191
Jul 10 18:40:57.360778 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:57.360748    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/0.log"
Jul 10 18:40:57.361072 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:57.360788    2561 generic.go:356] "Generic (PLEG): container finished" podID="8cd32a9e-c8d4-4e46-8737-e786070a64f6" containerID="311fc593cf4b2060812d1666b0a4d4e88fc82b9bb5fd481a3311d9182f45881d" exitCode=255
Jul 10 18:40:57.361072 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:57.360882    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-wfk4b" event={"ID":"8cd32a9e-c8d4-4e46-8737-e786070a64f6","Type":"ContainerDied","Data":"311fc593cf4b2060812d1666b0a4d4e88fc82b9bb5fd481a3311d9182f45881d"}
Jul 10 18:40:57.361209 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:57.361172    2561 scope.go:117] "RemoveContainer" containerID="311fc593cf4b2060812d1666b0a4d4e88fc82b9bb5fd481a3311d9182f45881d"
Jul 10 18:40:57.368907 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:57.368576    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-wfc8f" event={"ID":"1ac48545-82a6-41bc-87e8-890c729f1b94","Type":"ContainerStarted","Data":"1b1f98c065ff1ea526c7727031f1063f6cddae830139311bcc0201b019afc58e"}
Jul 10 18:40:57.368907 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:57.368674    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-wfc8f" event={"ID":"1ac48545-82a6-41bc-87e8-890c729f1b94","Type":"ContainerStarted","Data":"01810e7cfdf88076a633ea7100088198ecc841d1f7b32ef82a8b8ece568cf191"}
Jul 10 18:40:57.370261 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:57.370228    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2" event={"ID":"a4adda52-8024-4b86-a7f0-6f6ee4d9e731","Type":"ContainerStarted","Data":"20be0072753401e4fde731f1b25d7b84752240fa42e7dc962da97d08dc828eb5"}
Jul 10 18:40:57.371992 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:57.371972    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9" event={"ID":"27e2c236-1c19-48b0-ad1a-38d02e492b6c","Type":"ContainerStarted","Data":"62e27793672a13b0f3ef83bda59d26ba2d772f979f822337bb66151b021c1074"}
Jul 10 18:40:57.400181 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:57.400099    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9" podStartSLOduration=2.058861643 podStartE2EDuration="4.400056911s" totalImagesPullingTime="2.341195268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:57.399695203 +0000 UTC m=+80.873788089" watchObservedRunningTime="2026-07-10 18:40:57.400056911 +0000 UTC m=+80.874149797"
Jul 10 18:40:57.417825 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:57.417772    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-wfc8f" podStartSLOduration=1.417756722 podStartE2EDuration="1.417756722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:57.416357901 +0000 UTC m=+80.890450787" watchObservedRunningTime="2026-07-10 18:40:57.417756722 +0000 UTC m=+80.891849609"
Jul 10 18:40:57.436228 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:57.436101    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2" podStartSLOduration=1.6681436189999999 podStartE2EDuration="4.436066228s" totalImagesPullingTime="2.767922609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:57.434547125 +0000 UTC m=+80.908640034" watchObservedRunningTime="2026-07-10 18:40:57.436066228 +0000 UTC m=+80.910159114"
Jul 10 18:40:57.745385 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:57.745343    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:57.991839 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:57.991799    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:58.218788 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.218750    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:58.307483 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.307451    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nwzj8"]
Jul 10 18:40:58.310526 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.310509    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nwzj8"
Jul 10 18:40:58.312852 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.312832    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-82nmw\""
Jul 10 18:40:58.313024 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.312863    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 10 18:40:58.313682 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.313663    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 10 18:40:58.317000 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.316822    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:58.319966 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.319946    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nwzj8"]
Jul 10 18:40:58.376676 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.376648    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 18:40:58.377126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.377108    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/0.log"
Jul 10 18:40:58.377229 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.377151    2561 generic.go:356] "Generic (PLEG): container finished" podID="8cd32a9e-c8d4-4e46-8737-e786070a64f6" containerID="1db117343d4a739e30df140010aecaafa1cf0537f86912d0ce5e93b231eea760" exitCode=255
Jul 10 18:40:58.377287 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.377252    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-wfk4b" event={"ID":"8cd32a9e-c8d4-4e46-8737-e786070a64f6","Type":"ContainerDied","Data":"1db117343d4a739e30df140010aecaafa1cf0537f86912d0ce5e93b231eea760"}
Jul 10 18:40:58.377340 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.377305    2561 scope.go:117] "RemoveContainer" containerID="311fc593cf4b2060812d1666b0a4d4e88fc82b9bb5fd481a3311d9182f45881d"
Jul 10 18:40:58.377520 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.377501    2561 scope.go:117] "RemoveContainer" containerID="1db117343d4a739e30df140010aecaafa1cf0537f86912d0ce5e93b231eea760"
Jul 10 18:40:58.377753 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:58.377716    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-wfk4b_openshift-console-operator(8cd32a9e-c8d4-4e46-8737-e786070a64f6)\"" pod="openshift-console-operator/console-operator-7866497494-wfk4b" podUID="8cd32a9e-c8d4-4e46-8737-e786070a64f6"
Jul 10 18:40:58.399690 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.399658    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:58.402740 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.402719    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8b4l\" (UniqueName: \"kubernetes.io/projected/d1efc208-97e2-4f83-867c-084b1863357b-kube-api-access-w8b4l\") pod \"migrator-7f59b54f6c-nwzj8\" (UID: \"d1efc208-97e2-4f83-867c-084b1863357b\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nwzj8"
Jul 10 18:40:58.484095 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.483985    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:58.504274 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.504235    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8b4l\" (UniqueName: \"kubernetes.io/projected/d1efc208-97e2-4f83-867c-084b1863357b-kube-api-access-w8b4l\") pod \"migrator-7f59b54f6c-nwzj8\" (UID: \"d1efc208-97e2-4f83-867c-084b1863357b\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nwzj8"
Jul 10 18:40:58.512525 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.512489    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8b4l\" (UniqueName: \"kubernetes.io/projected/d1efc208-97e2-4f83-867c-084b1863357b-kube-api-access-w8b4l\") pod \"migrator-7f59b54f6c-nwzj8\" (UID: \"d1efc208-97e2-4f83-867c-084b1863357b\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nwzj8"
Jul 10 18:40:58.619726 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.619691    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nwzj8"
Jul 10 18:40:58.750376 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:58.750188    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nwzj8"]
Jul 10 18:40:58.752776 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:40:58.752746    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1efc208_97e2_4f83_867c_084b1863357b.slice/crio-a9b391797ff557b14f61ff1ef18113f149d52fc3ef46f2bd6c34316c222132e7 WatchSource:0}: Error finding container a9b391797ff557b14f61ff1ef18113f149d52fc3ef46f2bd6c34316c222132e7: Status 404 returned error can't find the container with id a9b391797ff557b14f61ff1ef18113f149d52fc3ef46f2bd6c34316c222132e7
Jul 10 18:40:59.016804 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.016703    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:59.118869 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.118834    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:59.382939 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.382762    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 18:40:59.383242 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.383221    2561 scope.go:117] "RemoveContainer" containerID="1db117343d4a739e30df140010aecaafa1cf0537f86912d0ce5e93b231eea760"
Jul 10 18:40:59.383507 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:59.383446    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-wfk4b_openshift-console-operator(8cd32a9e-c8d4-4e46-8737-e786070a64f6)\"" pod="openshift-console-operator/console-operator-7866497494-wfk4b" podUID="8cd32a9e-c8d4-4e46-8737-e786070a64f6"
Jul 10 18:40:59.384346 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.384305    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nwzj8" event={"ID":"d1efc208-97e2-4f83-867c-084b1863357b","Type":"ContainerStarted","Data":"a9b391797ff557b14f61ff1ef18113f149d52fc3ef46f2bd6c34316c222132e7"}
Jul 10 18:40:59.396015 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.393936    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dbkh5"]
Jul 10 18:40:59.404216 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.404188    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:40:59.410248 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.410211    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 10 18:40:59.410369 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.410260    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 10 18:40:59.411165 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.410951    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 10 18:40:59.411508 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.411351    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 10 18:40:59.413758 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.413594    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nfnkx\""
Jul 10 18:40:59.440135 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.440059    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dbkh5"]
Jul 10 18:40:59.512426 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.512385    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68b929ae-2f98-46da-b893-82064267e801-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:40:59.512630 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.512436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dztgk\" (UniqueName: \"kubernetes.io/projected/68b929ae-2f98-46da-b893-82064267e801-kube-api-access-dztgk\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:40:59.512630 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.512519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68b929ae-2f98-46da-b893-82064267e801-data-volume\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:40:59.512630 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.512590    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:40:59.512794 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.512757    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68b929ae-2f98-46da-b893-82064267e801-crio-socket\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:40:59.550557 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.550519    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:59.613918 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.613882    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68b929ae-2f98-46da-b893-82064267e801-crio-socket\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:40:59.614118 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.613930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68b929ae-2f98-46da-b893-82064267e801-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:40:59.614118 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.613957    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dztgk\" (UniqueName: \"kubernetes.io/projected/68b929ae-2f98-46da-b893-82064267e801-kube-api-access-dztgk\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:40:59.614118 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.613994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68b929ae-2f98-46da-b893-82064267e801-data-volume\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:40:59.614118 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.614026    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:40:59.614118 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.614025    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68b929ae-2f98-46da-b893-82064267e801-crio-socket\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:40:59.614118 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:59.614041    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:40:59.614443 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:59.614128    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68b929ae-2f98-46da-b893-82064267e801-kube-rbac-proxy-cm podName:68b929ae-2f98-46da-b893-82064267e801 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:00.114108311 +0000 UTC m=+83.588201194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68b929ae-2f98-46da-b893-82064267e801-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dbkh5" (UID: "68b929ae-2f98-46da-b893-82064267e801") : configmap "kube-rbac-proxy" not found
Jul 10 18:40:59.614443 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:59.614299    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:40:59.614443 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:40:59.614379    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls podName:68b929ae-2f98-46da-b893-82064267e801 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:00.114359603 +0000 UTC m=+83.588452477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dbkh5" (UID: "68b929ae-2f98-46da-b893-82064267e801") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:40:59.614443 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.614404    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68b929ae-2f98-46da-b893-82064267e801-data-volume\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:40:59.633510 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.633436    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dztgk\" (UniqueName: \"kubernetes.io/projected/68b929ae-2f98-46da-b893-82064267e801-kube-api-access-dztgk\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:40:59.670644 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:40:59.670609    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:00.117707 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:00.117671    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68b929ae-2f98-46da-b893-82064267e801-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:41:00.118117 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:00.117724    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:41:00.118117 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:00.117804    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:41:00.118117 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:00.117835    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:00.118117 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:00.117872    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68b929ae-2f98-46da-b893-82064267e801-kube-rbac-proxy-cm podName:68b929ae-2f98-46da-b893-82064267e801 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:01.117856127 +0000 UTC m=+84.591949014 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68b929ae-2f98-46da-b893-82064267e801-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dbkh5" (UID: "68b929ae-2f98-46da-b893-82064267e801") : configmap "kube-rbac-proxy" not found
Jul 10 18:41:00.118117 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:00.117886    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls podName:68b929ae-2f98-46da-b893-82064267e801 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:01.117880034 +0000 UTC m=+84.591972897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dbkh5" (UID: "68b929ae-2f98-46da-b893-82064267e801") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:00.149539 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:00.149455    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:00.240669 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:00.240637    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:00.390010 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:00.389979    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nwzj8" event={"ID":"d1efc208-97e2-4f83-867c-084b1863357b","Type":"ContainerStarted","Data":"24d39095b843d88ac5d36179c7dade5c8b8f733332e07da52d636e99465e4a1a"}
Jul 10 18:41:00.390191 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:00.390033    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nwzj8" event={"ID":"d1efc208-97e2-4f83-867c-084b1863357b","Type":"ContainerStarted","Data":"ab965d05be9a359fd61b939f70d7703c483f012f6ab89996c4ecdd73509aa88e"}
Jul 10 18:41:00.414446 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:00.414393    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nwzj8" podStartSLOduration=1.3734219730000001 podStartE2EDuration="2.414375437s" totalImagesPullingTime="1.040953464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:00.413881208 +0000 UTC m=+83.887974094" watchObservedRunningTime="2026-07-10 18:41:00.414375437 +0000 UTC m=+83.888468322"
Jul 10 18:41:00.661982 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:00.661947    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:00.801534 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:00.801447    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:01.126470 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:01.126378    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68b929ae-2f98-46da-b893-82064267e801-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:41:01.126470 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:01.126432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:41:01.126873 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:01.126501    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:41:01.126873 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:01.126549    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:01.126873 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:01.126572    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68b929ae-2f98-46da-b893-82064267e801-kube-rbac-proxy-cm podName:68b929ae-2f98-46da-b893-82064267e801 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:03.126556042 +0000 UTC m=+86.600648904 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68b929ae-2f98-46da-b893-82064267e801-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dbkh5" (UID: "68b929ae-2f98-46da-b893-82064267e801") : configmap "kube-rbac-proxy" not found
Jul 10 18:41:01.126873 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:01.126589    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls podName:68b929ae-2f98-46da-b893-82064267e801 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:03.126582851 +0000 UTC m=+86.600675714 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dbkh5" (UID: "68b929ae-2f98-46da-b893-82064267e801") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:01.267250 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:01.267214    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:02.181444 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:02.181419    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nq2tz_d4473d8a-7b80-4430-9006-674467a52709/dns-node-resolver/0.log"
Jul 10 18:41:02.880309 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:02.880277    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:02.976821 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:02.976793    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2tjml_3f0e0679-451d-492b-a83b-8c7662d459a7/node-ca/0.log"
Jul 10 18:41:02.981479 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:02.981451    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:03.144042 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:03.143952    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68b929ae-2f98-46da-b893-82064267e801-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:41:03.144042 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:03.144006    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:41:03.144301 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:03.144100    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:41:03.144301 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:03.144122    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:03.144301 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:03.144183    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68b929ae-2f98-46da-b893-82064267e801-kube-rbac-proxy-cm podName:68b929ae-2f98-46da-b893-82064267e801 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:07.144157621 +0000 UTC m=+90.618250484 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68b929ae-2f98-46da-b893-82064267e801-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dbkh5" (UID: "68b929ae-2f98-46da-b893-82064267e801") : configmap "kube-rbac-proxy" not found
Jul 10 18:41:03.144301 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:03.144201    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls podName:68b929ae-2f98-46da-b893-82064267e801 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:07.144194206 +0000 UTC m=+90.618287069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dbkh5" (UID: "68b929ae-2f98-46da-b893-82064267e801") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:03.549875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:03.549831    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-wfk4b"
Jul 10 18:41:03.550296 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:03.549926    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-wfk4b"
Jul 10 18:41:03.550342 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:03.550296    2561 scope.go:117] "RemoveContainer" containerID="1db117343d4a739e30df140010aecaafa1cf0537f86912d0ce5e93b231eea760"
Jul 10 18:41:03.550462 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:03.550440    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-wfk4b_openshift-console-operator(8cd32a9e-c8d4-4e46-8737-e786070a64f6)\"" pod="openshift-console-operator/console-operator-7866497494-wfk4b" podUID="8cd32a9e-c8d4-4e46-8737-e786070a64f6"
Jul 10 18:41:03.979562 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:03.979530    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cx9r7_91ec32f4-5c41-4dea-a301-97256b26976d/serve-healthcheck-canary/0.log"
Jul 10 18:41:04.221463 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:04.221428    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:04.376577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:04.376498    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-nwzj8_d1efc208-97e2-4f83-867c-084b1863357b/migrator/0.log"
Jul 10 18:41:04.576594 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:04.576568    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-nwzj8_d1efc208-97e2-4f83-867c-084b1863357b/graceful-termination/0.log"
Jul 10 18:41:04.779025 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:04.778991    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-v7qt9_27e2c236-1c19-48b0-ad1a-38d02e492b6c/kube-storage-version-migrator-operator/0.log"
Jul 10 18:41:05.404311 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:05.404277    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:05.497635 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:05.497597    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:05.594366 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:05.594333    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:05.695316 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:05.695219    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:06.265227 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:06.265194    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:06.452008 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:06.451969    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:07.181759 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:07.181709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68b929ae-2f98-46da-b893-82064267e801-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:41:07.181759 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:07.181776    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:41:07.182282 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:07.181847    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:41:07.182282 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:07.181904    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:07.182282 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:07.181934    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68b929ae-2f98-46da-b893-82064267e801-kube-rbac-proxy-cm podName:68b929ae-2f98-46da-b893-82064267e801 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:15.181911398 +0000 UTC m=+98.656004277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68b929ae-2f98-46da-b893-82064267e801-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dbkh5" (UID: "68b929ae-2f98-46da-b893-82064267e801") : configmap "kube-rbac-proxy" not found
Jul 10 18:41:07.182282 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:07.181957    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls podName:68b929ae-2f98-46da-b893-82064267e801 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:15.181946671 +0000 UTC m=+98.656039535 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dbkh5" (UID: "68b929ae-2f98-46da-b893-82064267e801") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:07.250221 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:07.250184    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:07.664445 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:07.664409    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:08.061176 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:08.061066    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:08.451458 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:08.451422    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:08.657749 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:08.657700    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:09.048269 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:09.048236    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:09.188578 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:09.188541    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:09.291306 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:09.291274    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:09.846697 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:09.846669    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:10.046849 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:10.046819    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:10.104991 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:10.104902    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:10.443427 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:10.443396    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:10.639100 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:10.639049    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:11.078128 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:11.078098    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:11.343179 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:11.343071    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:11.493090 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:11.493054    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:11.844109 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:11.844057    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:12.245683 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:12.245654    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:12.454885 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:12.454854    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:12.646481 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:12.646396    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:12.849518 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:12.849487    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:13.164907 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:13.164868    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:13.843632 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:13.843600    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:14.050328 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:14.050296    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:14.446969 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:14.446924    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:14.643379 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:14.643351    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:15.049283 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:15.049246    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:15.251041 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:15.250998    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:41:15.251306 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:15.251125    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68b929ae-2f98-46da-b893-82064267e801-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:41:15.251306 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:15.251178    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:15.251306 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:15.251268    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls podName:68b929ae-2f98-46da-b893-82064267e801 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:31.251251521 +0000 UTC m=+114.725344387 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dbkh5" (UID: "68b929ae-2f98-46da-b893-82064267e801") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:15.251685 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:15.251668    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68b929ae-2f98-46da-b893-82064267e801-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:41:15.440254 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:15.440201    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:15.849401 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:15.849319    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:16.038794 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:16.038760    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:16.259414 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:16.259372    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:41:16.259414 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:16.259417    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:41:16.261747 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:16.261710    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/560b6122-2684-461d-a4e2-34ef3351b1bc-metrics-tls\") pod \"dns-default-tlhxg\" (UID: \"560b6122-2684-461d-a4e2-34ef3351b1bc\") " pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:41:16.261859 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:16.261800    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls\") pod \"image-registry-d84bdf484-l56g2\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") " pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:41:16.549503 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:16.549414    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nkq5k\""
Jul 10 18:41:16.557630 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:16.557607    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:41:16.562486 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:16.562468    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4x2b4\""
Jul 10 18:41:16.570821 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:16.570787    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:41:16.701356 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:16.701326    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d84bdf484-l56g2"]
Jul 10 18:41:16.705038 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:41:16.705014    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2816d5_3984_4084_9706_111c92941084.slice/crio-ebf04fab9acc7afb2abe53123da9c4abe852be4053059f0971182c71f3e37352 WatchSource:0}: Error finding container ebf04fab9acc7afb2abe53123da9c4abe852be4053059f0971182c71f3e37352: Status 404 returned error can't find the container with id ebf04fab9acc7afb2abe53123da9c4abe852be4053059f0971182c71f3e37352
Jul 10 18:41:16.718926 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:16.718896    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tlhxg"]
Jul 10 18:41:16.722032 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:41:16.722004    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560b6122_2684_461d_a4e2_34ef3351b1bc.slice/crio-853ce5d90a10626a48ab6925cf4f7758154995b0416b512d93a63dfc7efb6f2b WatchSource:0}: Error finding container 853ce5d90a10626a48ab6925cf4f7758154995b0416b512d93a63dfc7efb6f2b: Status 404 returned error can't find the container with id 853ce5d90a10626a48ab6925cf4f7758154995b0416b512d93a63dfc7efb6f2b
Jul 10 18:41:17.446751 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:17.446709    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d84bdf484-l56g2" event={"ID":"bd2816d5-3984-4084-9706-111c92941084","Type":"ContainerStarted","Data":"9a28f9141b34959b3ad678cb2ca33ce906c984aa29dc132c81acea9988f2c1ff"}
Jul 10 18:41:17.447248 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:17.446767    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d84bdf484-l56g2" event={"ID":"bd2816d5-3984-4084-9706-111c92941084","Type":"ContainerStarted","Data":"ebf04fab9acc7afb2abe53123da9c4abe852be4053059f0971182c71f3e37352"}
Jul 10 18:41:17.447248 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:17.446852    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:41:17.448108 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:17.448068    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tlhxg" event={"ID":"560b6122-2684-461d-a4e2-34ef3351b1bc","Type":"ContainerStarted","Data":"853ce5d90a10626a48ab6925cf4f7758154995b0416b512d93a63dfc7efb6f2b"}
Jul 10 18:41:17.472498 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:17.472447    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d84bdf484-l56g2" podStartSLOduration=99.472430944 podStartE2EDuration="1m39.472430944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:17.472428411 +0000 UTC m=+100.946521298" watchObservedRunningTime="2026-07-10 18:41:17.472430944 +0000 UTC m=+100.946523829"
Jul 10 18:41:19.066853 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.066825    2561 scope.go:117] "RemoveContainer" containerID="1db117343d4a739e30df140010aecaafa1cf0537f86912d0ce5e93b231eea760"
Jul 10 18:41:19.329830 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.329749    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2pkwv"
Jul 10 18:41:19.461604 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.461570    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tlhxg" event={"ID":"560b6122-2684-461d-a4e2-34ef3351b1bc","Type":"ContainerStarted","Data":"c4185cd9d1a6c646d84538cfc23490a9617be54737c0b7869595bb6055951e6a"}
Jul 10 18:41:19.461795 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.461647    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tlhxg" event={"ID":"560b6122-2684-461d-a4e2-34ef3351b1bc","Type":"ContainerStarted","Data":"f9dd639c0e4fa92d92ba7d80805834a6369ea0a75a793339cf861eea430f53c6"}
Jul 10 18:41:19.461795 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.461688    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:41:19.463504 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.463487    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 18:41:19.463612 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.463543    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-wfk4b" event={"ID":"8cd32a9e-c8d4-4e46-8737-e786070a64f6","Type":"ContainerStarted","Data":"69a88e980f380fecabea7a13e3d000733329b88e9830ff5f357110c637e02162"}
Jul 10 18:41:19.463802 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.463787    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-wfk4b"
Jul 10 18:41:19.481395 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.481343    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tlhxg" podStartSLOduration=65.384415368 podStartE2EDuration="1m7.481329654s" totalImagesPullingTime="2.096914286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:19.481038773 +0000 UTC m=+102.955131677" watchObservedRunningTime="2026-07-10 18:41:19.481329654 +0000 UTC m=+102.955422538"
Jul 10 18:41:19.499411 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.499359    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-wfk4b" podStartSLOduration=23.746300886 podStartE2EDuration="26.499343829s" totalImagesPullingTime="2.753042943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:19.4984794 +0000 UTC m=+102.972572286" watchObservedRunningTime="2026-07-10 18:41:19.499343829 +0000 UTC m=+102.973436771"
Jul 10 18:41:19.932784 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.932752    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g"]
Jul 10 18:41:19.951150 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.951118    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g"
Jul 10 18:41:19.957232 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.957207    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 10 18:41:19.958408 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.957887    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 10 18:41:19.958492 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.958434    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 10 18:41:19.958790 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.958761    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 10 18:41:19.959294 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.959274    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g"]
Jul 10 18:41:19.987204 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.987172    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d89fe58f-7ae0-445c-b915-7f65a9c75560-klusterlet-config\") pod \"klusterlet-addon-workmgr-f6b558cc8-szd9g\" (UID: \"d89fe58f-7ae0-445c-b915-7f65a9c75560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g"
Jul 10 18:41:19.987363 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.987235    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnxw4\" (UniqueName: \"kubernetes.io/projected/d89fe58f-7ae0-445c-b915-7f65a9c75560-kube-api-access-wnxw4\") pod \"klusterlet-addon-workmgr-f6b558cc8-szd9g\" (UID: \"d89fe58f-7ae0-445c-b915-7f65a9c75560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g"
Jul 10 18:41:19.987429 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:19.987353    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d89fe58f-7ae0-445c-b915-7f65a9c75560-tmp\") pod \"klusterlet-addon-workmgr-f6b558cc8-szd9g\" (UID: \"d89fe58f-7ae0-445c-b915-7f65a9c75560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g"
Jul 10 18:41:20.007471 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.007442    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-wfk4b"
Jul 10 18:41:20.088546 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.088516    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d89fe58f-7ae0-445c-b915-7f65a9c75560-tmp\") pod \"klusterlet-addon-workmgr-f6b558cc8-szd9g\" (UID: \"d89fe58f-7ae0-445c-b915-7f65a9c75560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g"
Jul 10 18:41:20.088974 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.088575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d89fe58f-7ae0-445c-b915-7f65a9c75560-klusterlet-config\") pod \"klusterlet-addon-workmgr-f6b558cc8-szd9g\" (UID: \"d89fe58f-7ae0-445c-b915-7f65a9c75560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g"
Jul 10 18:41:20.088974 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.088676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnxw4\" (UniqueName: \"kubernetes.io/projected/d89fe58f-7ae0-445c-b915-7f65a9c75560-kube-api-access-wnxw4\") pod \"klusterlet-addon-workmgr-f6b558cc8-szd9g\" (UID: \"d89fe58f-7ae0-445c-b915-7f65a9c75560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g"
Jul 10 18:41:20.088974 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.088927    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d89fe58f-7ae0-445c-b915-7f65a9c75560-tmp\") pod \"klusterlet-addon-workmgr-f6b558cc8-szd9g\" (UID: \"d89fe58f-7ae0-445c-b915-7f65a9c75560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g"
Jul 10 18:41:20.091558 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.091529    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d89fe58f-7ae0-445c-b915-7f65a9c75560-klusterlet-config\") pod \"klusterlet-addon-workmgr-f6b558cc8-szd9g\" (UID: \"d89fe58f-7ae0-445c-b915-7f65a9c75560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g"
Jul 10 18:41:20.099302 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.099266    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnxw4\" (UniqueName: \"kubernetes.io/projected/d89fe58f-7ae0-445c-b915-7f65a9c75560-kube-api-access-wnxw4\") pod \"klusterlet-addon-workmgr-f6b558cc8-szd9g\" (UID: \"d89fe58f-7ae0-445c-b915-7f65a9c75560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g"
Jul 10 18:41:20.126866 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.126833    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:20.207257 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.207175    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-5vtgc"]
Jul 10 18:41:20.226991 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.226960    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:20.232784 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.232762    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-5vtgc"]
Jul 10 18:41:20.232924 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.232854    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-5vtgc"
Jul 10 18:41:20.236179 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.236157    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-rc9w9\""
Jul 10 18:41:20.236796 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.236777    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 10 18:41:20.236890 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.236874    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 10 18:41:20.262455 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.262421    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g"
Jul 10 18:41:20.292669 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.292631    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb926\" (UniqueName: \"kubernetes.io/projected/2732d02d-0e1b-4073-aba7-e668344351c9-kube-api-access-fb926\") pod \"downloads-68c9cbb497-5vtgc\" (UID: \"2732d02d-0e1b-4073-aba7-e668344351c9\") " pod="openshift-console/downloads-68c9cbb497-5vtgc"
Jul 10 18:41:20.310148 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.310118    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:20.393300 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.393274    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb926\" (UniqueName: \"kubernetes.io/projected/2732d02d-0e1b-4073-aba7-e668344351c9-kube-api-access-fb926\") pod \"downloads-68c9cbb497-5vtgc\" (UID: \"2732d02d-0e1b-4073-aba7-e668344351c9\") " pod="openshift-console/downloads-68c9cbb497-5vtgc"
Jul 10 18:41:20.411753 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.411722    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb926\" (UniqueName: \"kubernetes.io/projected/2732d02d-0e1b-4073-aba7-e668344351c9-kube-api-access-fb926\") pod \"downloads-68c9cbb497-5vtgc\" (UID: \"2732d02d-0e1b-4073-aba7-e668344351c9\") " pod="openshift-console/downloads-68c9cbb497-5vtgc"
Jul 10 18:41:20.412524 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.412485    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g"]
Jul 10 18:41:20.413961 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.413922    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:20.415273 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:41:20.415247    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89fe58f_7ae0_445c_b915_7f65a9c75560.slice/crio-4d92bcae1aa2933795cde1b17a925d72926d6b8a25171d09a0cae0c82f55352d WatchSource:0}: Error finding container 4d92bcae1aa2933795cde1b17a925d72926d6b8a25171d09a0cae0c82f55352d: Status 404 returned error can't find the container with id 4d92bcae1aa2933795cde1b17a925d72926d6b8a25171d09a0cae0c82f55352d
Jul 10 18:41:20.467955 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.467856    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g" event={"ID":"d89fe58f-7ae0-445c-b915-7f65a9c75560","Type":"ContainerStarted","Data":"4d92bcae1aa2933795cde1b17a925d72926d6b8a25171d09a0cae0c82f55352d"}
Jul 10 18:41:20.542805 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.542769    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-5vtgc"
Jul 10 18:41:20.547248 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.547219    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:20.697132 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.696954    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-5vtgc"]
Jul 10 18:41:20.699538 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:41:20.699515    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2732d02d_0e1b_4073_aba7_e668344351c9.slice/crio-20920a2879a209f90ec6eabd37c7f54be2c9b06c1813e5f9ee657f37f8c6f98d WatchSource:0}: Error finding container 20920a2879a209f90ec6eabd37c7f54be2c9b06c1813e5f9ee657f37f8c6f98d: Status 404 returned error can't find the container with id 20920a2879a209f90ec6eabd37c7f54be2c9b06c1813e5f9ee657f37f8c6f98d
Jul 10 18:41:20.720016 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.719938    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:20.907567 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:20.907534    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:21.008963 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:21.008869    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:21.474048 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:21.474011    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-5vtgc" event={"ID":"2732d02d-0e1b-4073-aba7-e668344351c9","Type":"ContainerStarted","Data":"20920a2879a209f90ec6eabd37c7f54be2c9b06c1813e5f9ee657f37f8c6f98d"}
Jul 10 18:41:21.892803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:21.892724    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:22.105829 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:22.105789    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:22.203941 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:22.203897    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:22.411823 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:22.411766    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:22.507563 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:22.507460    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:22.607930 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:22.607881    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:23.514949 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:23.514884    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:23.576427 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:23.576360    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:24.391760 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:24.391724    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:24.450646 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:24.450609    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:25.494250 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:25.494157    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g" event={"ID":"d89fe58f-7ae0-445c-b915-7f65a9c75560","Type":"ContainerStarted","Data":"936f380dc4e5178cdc0cfe55c11e6cf19c4c92d9eb6f54417dfd4c1b19c8c6fe"}
Jul 10 18:41:25.494673 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:25.494403    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g"
Jul 10 18:41:25.496322 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:25.496301    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g"
Jul 10 18:41:25.512834 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:25.512780    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f6b558cc8-szd9g" podStartSLOduration=1.754253517 podStartE2EDuration="6.512765854s" totalImagesPullingTime="4.758512337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:25.511658039 +0000 UTC m=+108.985750923" watchObservedRunningTime="2026-07-10 18:41:25.512765854 +0000 UTC m=+108.986858738"
Jul 10 18:41:28.325652 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:28.325610    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:28.418027 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:28.417983    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:28.524987 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:28.524893    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:28.793975 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:28.793936    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:29.146933 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:29.146854    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:29.470240 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:29.470212    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tlhxg"
Jul 10 18:41:30.532665 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:30.532629    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:30.633897 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:30.633861    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:31.292266 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:31.292226    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:41:31.294953 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:31.294927    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b929ae-2f98-46da-b893-82064267e801-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dbkh5\" (UID: \"68b929ae-2f98-46da-b893-82064267e801\") " pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:41:31.515896 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:31.515864    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dbkh5"
Jul 10 18:41:31.589387 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:31.589305    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:31.680641 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:31.680604    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:31.769231 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:31.769192    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:31.841113 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:31.840926    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:32.318985 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:32.318945    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:34.382688 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:34.382652    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:34.450644 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:34.450607    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:36.562587 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:36.562545    2561 patch_prober.go:28] interesting pod/image-registry-d84bdf484-l56g2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 10 18:41:36.563024 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:36.562609    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-d84bdf484-l56g2" podUID="bd2816d5-3984-4084-9706-111c92941084" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:41:37.167223 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:37.167186    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:37.523107 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:37.523003    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:38.457210 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:38.457187    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:41:38.457582 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:38.457410    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dbkh5"]
Jul 10 18:41:38.463157 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:41:38.463131    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b929ae_2f98_46da_b893_82064267e801.slice/crio-63f1fa4038c806a9aa250036b69142d2e31201795acf0e7b843cf6c1783d7736 WatchSource:0}: Error finding container 63f1fa4038c806a9aa250036b69142d2e31201795acf0e7b843cf6c1783d7736: Status 404 returned error can't find the container with id 63f1fa4038c806a9aa250036b69142d2e31201795acf0e7b843cf6c1783d7736
Jul 10 18:41:38.543644 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:38.543604    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-5vtgc" event={"ID":"2732d02d-0e1b-4073-aba7-e668344351c9","Type":"ContainerStarted","Data":"c901f221eac95a993c3adc64763f695644822eeeea26d687dc72f1e247de4e75"}
Jul 10 18:41:38.543836 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:38.543706    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-5vtgc"
Jul 10 18:41:38.544876 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:38.544850    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dbkh5" event={"ID":"68b929ae-2f98-46da-b893-82064267e801","Type":"ContainerStarted","Data":"63f1fa4038c806a9aa250036b69142d2e31201795acf0e7b843cf6c1783d7736"}
Jul 10 18:41:38.545298 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:38.545276    2561 patch_prober.go:28] interesting pod/downloads-68c9cbb497-5vtgc container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.16:8080/\": dial tcp 10.132.0.16:8080: connect: connection refused" start-of-body=
Jul 10 18:41:38.545378 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:38.545329    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-68c9cbb497-5vtgc" podUID="2732d02d-0e1b-4073-aba7-e668344351c9" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.16:8080/\": dial tcp 10.132.0.16:8080: connect: connection refused"
Jul 10 18:41:38.563626 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:38.563571    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-5vtgc" podStartSLOduration=0.83893382 podStartE2EDuration="18.563556859s" totalImagesPullingTime="17.724623039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:38.563311688 +0000 UTC m=+122.037404574" watchObservedRunningTime="2026-07-10 18:41:38.563556859 +0000 UTC m=+122.037649744"
Jul 10 18:41:38.751045 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:38.751006    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:38.848131 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:38.847978    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:39.141250 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:39.141158    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:39.240923 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:39.240842    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:39.562442 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:39.562277    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-5vtgc"
Jul 10 18:41:39.799972 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:39.799694    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:39.906101 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:39.906031    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:40.002213 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.002170    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:40.153635 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.153598    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:40.212055 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.211572    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp"]
Jul 10 18:41:40.243483 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.243402    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp"
Jul 10 18:41:40.248480 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.248399    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp"]
Jul 10 18:41:40.250178 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.249715    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 10 18:41:40.250178 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.249886    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5k6dd\""
Jul 10 18:41:40.250178 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.249982    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 10 18:41:40.250178 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.250159    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 10 18:41:40.250509 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.250402    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 10 18:41:40.250811 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.250649    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 10 18:41:40.265947 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.265859    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:40.305746 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.305685    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-f47gv"]
Jul 10 18:41:40.329809 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.329773    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.332802 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.332604    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 10 18:41:40.333906 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.333880    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 10 18:41:40.334166 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.334139    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 10 18:41:40.334594 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.334573    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5ggl7\""
Jul 10 18:41:40.383865 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.383573    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fb2c50d-7009-417b-927b-bbb8e45c94e0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-d9kxp\" (UID: \"1fb2c50d-7009-417b-927b-bbb8e45c94e0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp"
Jul 10 18:41:40.383865 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.383628    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fb2c50d-7009-417b-927b-bbb8e45c94e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-d9kxp\" (UID: \"1fb2c50d-7009-417b-927b-bbb8e45c94e0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp"
Jul 10 18:41:40.383865 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.383678    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4nnc\" (UniqueName: \"kubernetes.io/projected/1fb2c50d-7009-417b-927b-bbb8e45c94e0-kube-api-access-n4nnc\") pod \"openshift-state-metrics-84bd68b896-d9kxp\" (UID: \"1fb2c50d-7009-417b-927b-bbb8e45c94e0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp"
Jul 10 18:41:40.383865 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.383710    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fb2c50d-7009-417b-927b-bbb8e45c94e0-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-d9kxp\" (UID: \"1fb2c50d-7009-417b-927b-bbb8e45c94e0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp"
Jul 10 18:41:40.391684 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.391599    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:40.484612 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.484517    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebef81d2-c0d1-4894-9ca0-391f361b659b-root\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.484612 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.484569    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebef81d2-c0d1-4894-9ca0-391f361b659b-node-exporter-textfile\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.484612 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.484603    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fb2c50d-7009-417b-927b-bbb8e45c94e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-d9kxp\" (UID: \"1fb2c50d-7009-417b-927b-bbb8e45c94e0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp"
Jul 10 18:41:40.484875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.484694    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebef81d2-c0d1-4894-9ca0-391f361b659b-metrics-client-ca\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.484875 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:40.484710    2561 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 10 18:41:40.484875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.484739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebef81d2-c0d1-4894-9ca0-391f361b659b-node-exporter-accelerators-collector-config\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.484875 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:40.484776    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb2c50d-7009-417b-927b-bbb8e45c94e0-openshift-state-metrics-tls podName:1fb2c50d-7009-417b-927b-bbb8e45c94e0 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:40.984754445 +0000 UTC m=+124.458847330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1fb2c50d-7009-417b-927b-bbb8e45c94e0-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-d9kxp" (UID: "1fb2c50d-7009-417b-927b-bbb8e45c94e0") : secret "openshift-state-metrics-tls" not found
Jul 10 18:41:40.484875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.484867    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4nnc\" (UniqueName: \"kubernetes.io/projected/1fb2c50d-7009-417b-927b-bbb8e45c94e0-kube-api-access-n4nnc\") pod \"openshift-state-metrics-84bd68b896-d9kxp\" (UID: \"1fb2c50d-7009-417b-927b-bbb8e45c94e0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp"
Jul 10 18:41:40.485162 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.484901    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfkqm\" (UniqueName: \"kubernetes.io/projected/ebef81d2-c0d1-4894-9ca0-391f361b659b-kube-api-access-qfkqm\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.485162 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.484939    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fb2c50d-7009-417b-927b-bbb8e45c94e0-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-d9kxp\" (UID: \"1fb2c50d-7009-417b-927b-bbb8e45c94e0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp"
Jul 10 18:41:40.485162 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.484960    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebef81d2-c0d1-4894-9ca0-391f361b659b-node-exporter-tls\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.485162 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.485008    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebef81d2-c0d1-4894-9ca0-391f361b659b-node-exporter-wtmp\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.485162 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.485094    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebef81d2-c0d1-4894-9ca0-391f361b659b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.485162 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.485127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebef81d2-c0d1-4894-9ca0-391f361b659b-sys\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.485482 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.485165    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fb2c50d-7009-417b-927b-bbb8e45c94e0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-d9kxp\" (UID: \"1fb2c50d-7009-417b-927b-bbb8e45c94e0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp"
Jul 10 18:41:40.485786 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.485757    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fb2c50d-7009-417b-927b-bbb8e45c94e0-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-d9kxp\" (UID: \"1fb2c50d-7009-417b-927b-bbb8e45c94e0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp"
Jul 10 18:41:40.487877 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.487851    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fb2c50d-7009-417b-927b-bbb8e45c94e0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-d9kxp\" (UID: \"1fb2c50d-7009-417b-927b-bbb8e45c94e0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp"
Jul 10 18:41:40.498819 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.498795    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4nnc\" (UniqueName: \"kubernetes.io/projected/1fb2c50d-7009-417b-927b-bbb8e45c94e0-kube-api-access-n4nnc\") pod \"openshift-state-metrics-84bd68b896-d9kxp\" (UID: \"1fb2c50d-7009-417b-927b-bbb8e45c94e0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp"
Jul 10 18:41:40.503069 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.503041    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:40.586436 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.586402    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebef81d2-c0d1-4894-9ca0-391f361b659b-root\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.586896 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.586442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebef81d2-c0d1-4894-9ca0-391f361b659b-node-exporter-textfile\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.586896 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.586495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebef81d2-c0d1-4894-9ca0-391f361b659b-metrics-client-ca\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.586896 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.586525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebef81d2-c0d1-4894-9ca0-391f361b659b-node-exporter-accelerators-collector-config\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.586896 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.586533    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebef81d2-c0d1-4894-9ca0-391f361b659b-root\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.586896 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.586556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfkqm\" (UniqueName: \"kubernetes.io/projected/ebef81d2-c0d1-4894-9ca0-391f361b659b-kube-api-access-qfkqm\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.586896 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.586628    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebef81d2-c0d1-4894-9ca0-391f361b659b-node-exporter-tls\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.586896 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.586697    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebef81d2-c0d1-4894-9ca0-391f361b659b-node-exporter-wtmp\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.586896 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.586757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebef81d2-c0d1-4894-9ca0-391f361b659b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.586896 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.586784    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebef81d2-c0d1-4894-9ca0-391f361b659b-sys\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.586896 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.586876    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebef81d2-c0d1-4894-9ca0-391f361b659b-sys\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.587555 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.587150    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebef81d2-c0d1-4894-9ca0-391f361b659b-node-exporter-wtmp\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.588010 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.587967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebef81d2-c0d1-4894-9ca0-391f361b659b-metrics-client-ca\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.588651 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.588208    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebef81d2-c0d1-4894-9ca0-391f361b659b-node-exporter-textfile\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.588651 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.588605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebef81d2-c0d1-4894-9ca0-391f361b659b-node-exporter-accelerators-collector-config\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.590831 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.590786    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebef81d2-c0d1-4894-9ca0-391f361b659b-node-exporter-tls\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.593894 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.593623    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebef81d2-c0d1-4894-9ca0-391f361b659b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.596133 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.596109    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfkqm\" (UniqueName: \"kubernetes.io/projected/ebef81d2-c0d1-4894-9ca0-391f361b659b-kube-api-access-qfkqm\") pod \"node-exporter-f47gv\" (UID: \"ebef81d2-c0d1-4894-9ca0-391f361b659b\") " pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.642901 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.642644    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f47gv"
Jul 10 18:41:40.658128 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.657907    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:40.749959 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.749879    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:40.990747 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.990709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fb2c50d-7009-417b-927b-bbb8e45c94e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-d9kxp\" (UID: \"1fb2c50d-7009-417b-927b-bbb8e45c94e0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp"
Jul 10 18:41:40.993824 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:40.993792    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fb2c50d-7009-417b-927b-bbb8e45c94e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-d9kxp\" (UID: \"1fb2c50d-7009-417b-927b-bbb8e45c94e0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp"
Jul 10 18:41:41.117546 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.117463    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:41.161251 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.161205    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp"
Jul 10 18:41:41.261326 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.261246    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:41.278152 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.278117    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 10 18:41:41.307583 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.307548    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.314160 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.314133    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 10 18:41:41.314312 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.314165    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 10 18:41:41.314401 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.314386    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 10 18:41:41.314659 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.314612    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hxmg9\""
Jul 10 18:41:41.314839 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.314818    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 10 18:41:41.314998 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.314979    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 10 18:41:41.315145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.314388    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 10 18:41:41.315472 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.315454    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 10 18:41:41.315788 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.315771    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 10 18:41:41.322128 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.321344    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 10 18:41:41.325606 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.325571    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 10 18:41:41.395658 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.395572    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.395658 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.395645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b48ffef0-0728-4741-a349-a28e00ca1d0b-config-out\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.395865 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.395692    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.395865 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.395843    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddd24\" (UniqueName: \"kubernetes.io/projected/b48ffef0-0728-4741-a349-a28e00ca1d0b-kube-api-access-ddd24\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.396025 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.395897    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b48ffef0-0728-4741-a349-a28e00ca1d0b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.396025 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.395930    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b48ffef0-0728-4741-a349-a28e00ca1d0b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.396025 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.395968    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-config-volume\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.396025 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.396001    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b48ffef0-0728-4741-a349-a28e00ca1d0b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.396458 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.396032    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b48ffef0-0728-4741-a349-a28e00ca1d0b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.396458 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.396056    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.396458 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.396113    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.396458 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.396155    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.396458 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.396201    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-web-config\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.496879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.496841    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.497056 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.496902    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.497225 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.497193    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-web-config\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.497303 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.497249    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.497484 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.497456    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b48ffef0-0728-4741-a349-a28e00ca1d0b-config-out\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.497575 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.497501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.497575 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.497536    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddd24\" (UniqueName: \"kubernetes.io/projected/b48ffef0-0728-4741-a349-a28e00ca1d0b-kube-api-access-ddd24\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.497676 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.497573    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b48ffef0-0728-4741-a349-a28e00ca1d0b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.497676 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.497599    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b48ffef0-0728-4741-a349-a28e00ca1d0b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.497676 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.497636    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-config-volume\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.497676 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.497670    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b48ffef0-0728-4741-a349-a28e00ca1d0b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.497861 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.497699    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b48ffef0-0728-4741-a349-a28e00ca1d0b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.497861 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.497728    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.502406 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.500635    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b48ffef0-0728-4741-a349-a28e00ca1d0b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.502406 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.501111    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.502406 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.501390    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b48ffef0-0728-4741-a349-a28e00ca1d0b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.502406 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.502344    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b48ffef0-0728-4741-a349-a28e00ca1d0b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.502806 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.502778    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b48ffef0-0728-4741-a349-a28e00ca1d0b-config-out\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.505145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.505097    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-config-volume\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.505259 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.505175    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.505732 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.505710    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b48ffef0-0728-4741-a349-a28e00ca1d0b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.506781 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.506751    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-web-config\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.506913 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.506806    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.507907 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.507864    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.508389 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.508364    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.527151 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.527112    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddd24\" (UniqueName: \"kubernetes.io/projected/b48ffef0-0728-4741-a349-a28e00ca1d0b-kube-api-access-ddd24\") pod \"alertmanager-main-0\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.611304 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.611252    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:41.627228 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.627181    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:41:41.699699 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.699654    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d84bdf484-l56g2"]
Jul 10 18:41:41.712602 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.712332    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:41.824646 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.824610    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:41.924850 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:41.924818    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:42.359859 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:42.359823    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:42.463780 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:42.463719    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:42.985190 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:42.985154    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:43.152453 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.152415    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:43.153303 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.153260    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"]
Jul 10 18:41:43.175746 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.175716    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"]
Jul 10 18:41:43.175986 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.175863    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.180163 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.179913    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 10 18:41:43.180287 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.180167    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 10 18:41:43.180448 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.180400    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-625hh50qgd691\""
Jul 10 18:41:43.180671 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.180653    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 10 18:41:43.180772 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.180660    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 10 18:41:43.180891 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.180874    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 10 18:41:43.181067 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.181040    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hdbz6\""
Jul 10 18:41:43.256065 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.255957    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:43.316744 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.316565    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66182fd1-a634-4db8-b35e-d64b8632eadc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.316744 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.316624    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/66182fd1-a634-4db8-b35e-d64b8632eadc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.316744 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.316680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6glz2\" (UniqueName: \"kubernetes.io/projected/66182fd1-a634-4db8-b35e-d64b8632eadc-kube-api-access-6glz2\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.316744 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.316709    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66182fd1-a634-4db8-b35e-d64b8632eadc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.317117 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.316867    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/66182fd1-a634-4db8-b35e-d64b8632eadc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.317117 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.316916    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66182fd1-a634-4db8-b35e-d64b8632eadc-metrics-client-ca\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.317117 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.316947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/66182fd1-a634-4db8-b35e-d64b8632eadc-secret-thanos-querier-tls\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.317117 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.316973    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66182fd1-a634-4db8-b35e-d64b8632eadc-secret-grpc-tls\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.355600 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.355454    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:43.418100 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.418044    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66182fd1-a634-4db8-b35e-d64b8632eadc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.418306 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.418113    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/66182fd1-a634-4db8-b35e-d64b8632eadc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.418306 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.418178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6glz2\" (UniqueName: \"kubernetes.io/projected/66182fd1-a634-4db8-b35e-d64b8632eadc-kube-api-access-6glz2\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.418306 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.418210    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66182fd1-a634-4db8-b35e-d64b8632eadc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.418306 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.418298    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/66182fd1-a634-4db8-b35e-d64b8632eadc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.418574 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.418333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66182fd1-a634-4db8-b35e-d64b8632eadc-metrics-client-ca\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.418865 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.418832    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/66182fd1-a634-4db8-b35e-d64b8632eadc-secret-thanos-querier-tls\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.419267 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.418879    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66182fd1-a634-4db8-b35e-d64b8632eadc-secret-grpc-tls\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.419575 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.419383    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66182fd1-a634-4db8-b35e-d64b8632eadc-metrics-client-ca\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.422045 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.421997    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/66182fd1-a634-4db8-b35e-d64b8632eadc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.423119 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.423007    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66182fd1-a634-4db8-b35e-d64b8632eadc-secret-grpc-tls\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.423503 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.423455    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/66182fd1-a634-4db8-b35e-d64b8632eadc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.423961 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.423935    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/66182fd1-a634-4db8-b35e-d64b8632eadc-secret-thanos-querier-tls\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.424048 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.424033    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66182fd1-a634-4db8-b35e-d64b8632eadc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.424454 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.424430    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66182fd1-a634-4db8-b35e-d64b8632eadc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.432385 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.432356    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6glz2\" (UniqueName: \"kubernetes.io/projected/66182fd1-a634-4db8-b35e-d64b8632eadc-kube-api-access-6glz2\") pod \"thanos-querier-5dd7ccc499-69tw6\" (UID: \"66182fd1-a634-4db8-b35e-d64b8632eadc\") " pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.476530 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.476439    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:43.490345 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.490305    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:43.577501 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.577462    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:43.580915 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:41:43.580876    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebef81d2_c0d1_4894_9ca0_391f361b659b.slice/crio-86b894bceff37f641c4a0288a4906364d65986483e6534635d4f99c6e8a3f1f9 WatchSource:0}: Error finding container 86b894bceff37f641c4a0288a4906364d65986483e6534635d4f99c6e8a3f1f9: Status 404 returned error can't find the container with id 86b894bceff37f641c4a0288a4906364d65986483e6534635d4f99c6e8a3f1f9
Jul 10 18:41:43.680882 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.680743    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:43.795629 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.795509    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 10 18:41:43.800491 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.800460    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp"]
Jul 10 18:41:43.821397 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:43.821191    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"]
Jul 10 18:41:43.982566 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:41:43.982524    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48ffef0_0728_4741_a349_a28e00ca1d0b.slice/crio-969d2ec698b2ba4cb1f2e6f0dbcfeb15e6f74615b9801549c730abbebc19af4c WatchSource:0}: Error finding container 969d2ec698b2ba4cb1f2e6f0dbcfeb15e6f74615b9801549c730abbebc19af4c: Status 404 returned error can't find the container with id 969d2ec698b2ba4cb1f2e6f0dbcfeb15e6f74615b9801549c730abbebc19af4c
Jul 10 18:41:43.984556 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:41:43.984461    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb2c50d_7009_417b_927b_bbb8e45c94e0.slice/crio-00845850995fc86e7939c77c2ddf5cc57fb1ac175fe3ffcca10fa0c2be01d3fe WatchSource:0}: Error finding container 00845850995fc86e7939c77c2ddf5cc57fb1ac175fe3ffcca10fa0c2be01d3fe: Status 404 returned error can't find the container with id 00845850995fc86e7939c77c2ddf5cc57fb1ac175fe3ffcca10fa0c2be01d3fe
Jul 10 18:41:43.984743 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:41:43.984715    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66182fd1_a634_4db8_b35e_d64b8632eadc.slice/crio-947e8f95229f244734d757f47a19f23127f104c7d2e1a28dd1d182f78c2e774b WatchSource:0}: Error finding container 947e8f95229f244734d757f47a19f23127f104c7d2e1a28dd1d182f78c2e774b: Status 404 returned error can't find the container with id 947e8f95229f244734d757f47a19f23127f104c7d2e1a28dd1d182f78c2e774b
Jul 10 18:41:44.578006 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:44.577788    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp" event={"ID":"1fb2c50d-7009-417b-927b-bbb8e45c94e0","Type":"ContainerStarted","Data":"585d858cc4b39cfbe1d4b2c288b515971cd0326ecc53ab973d5ef8a2f1eae118"}
Jul 10 18:41:44.578006 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:44.577865    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp" event={"ID":"1fb2c50d-7009-417b-927b-bbb8e45c94e0","Type":"ContainerStarted","Data":"5aebabfe81e5306987579a57e1b102df5cb5175f006b878f0bc601eb850df0e7"}
Jul 10 18:41:44.578006 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:44.577898    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp" event={"ID":"1fb2c50d-7009-417b-927b-bbb8e45c94e0","Type":"ContainerStarted","Data":"00845850995fc86e7939c77c2ddf5cc57fb1ac175fe3ffcca10fa0c2be01d3fe"}
Jul 10 18:41:44.580834 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:44.580722    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b48ffef0-0728-4741-a349-a28e00ca1d0b","Type":"ContainerStarted","Data":"969d2ec698b2ba4cb1f2e6f0dbcfeb15e6f74615b9801549c730abbebc19af4c"}
Jul 10 18:41:44.584276 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:44.584246    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dbkh5" event={"ID":"68b929ae-2f98-46da-b893-82064267e801","Type":"ContainerStarted","Data":"517fa0b1e31c5b90e3673fcdc1d3c47c3a54ea051c4d573b162c320e8e20dfd1"}
Jul 10 18:41:44.587440 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:44.587417    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f47gv" event={"ID":"ebef81d2-c0d1-4894-9ca0-391f361b659b","Type":"ContainerStarted","Data":"86b894bceff37f641c4a0288a4906364d65986483e6534635d4f99c6e8a3f1f9"}
Jul 10 18:41:44.589203 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:44.589120    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6" event={"ID":"66182fd1-a634-4db8-b35e-d64b8632eadc","Type":"ContainerStarted","Data":"947e8f95229f244734d757f47a19f23127f104c7d2e1a28dd1d182f78c2e774b"}
Jul 10 18:41:44.875972 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:44.875876    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:44.987774 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:44.987735    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:45.098347 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:45.098309    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:46.528428 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.528366    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:46.530605 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.530578    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:41:46.566314 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.565296    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.574718 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.574606    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 10 18:41:46.574881 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.574741    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 10 18:41:46.575269 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.575052    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 10 18:41:46.575337 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.575308    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 10 18:41:46.575774 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.575739    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 10 18:41:46.575967 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.575914    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 10 18:41:46.576095 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.576009    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 10 18:41:46.576095 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.575914    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 10 18:41:46.576251 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.576154    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 10 18:41:46.582622 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.576930    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 10 18:41:46.582622 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.577001    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2nmhp\""
Jul 10 18:41:46.582622 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.577250    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 10 18:41:46.582622 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.577393    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 10 18:41:46.582622 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.577528    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-111d5q2uqf0p6\""
Jul 10 18:41:46.582622 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.577658    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 10 18:41:46.582622 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.579377    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:41:46.582622 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.581535    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 10 18:41:46.583275 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.583246    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 10 18:41:46.624265 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.624221    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:46.660274 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660158    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-config\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.660274 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660207    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc8df723-b571-4680-b235-ab8b02aa0b7a-config-out\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.660274 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660239    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.660604 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660316    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.660604 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660380    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.660604 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660485    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.660604 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660532    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.660604 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660598    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.660828 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660625    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.660828 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660650    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.660828 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660673    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.660828 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660696    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-web-config\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.660828 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660725    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.660828 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660742    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.660828 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660766    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.660828 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660803    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.661204 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660850    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvbqg\" (UniqueName: \"kubernetes.io/projected/bc8df723-b571-4680-b235-ab8b02aa0b7a-kube-api-access-gvbqg\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.661204 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660900    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc8df723-b571-4680-b235-ab8b02aa0b7a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.661204 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660953    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.661204 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.660986    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.762503 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.762461    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-config\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.762503 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.762507    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc8df723-b571-4680-b235-ab8b02aa0b7a-config-out\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.762749 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.762532    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.762749 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.762582    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.762749 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.762607    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.762749 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.762641    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.765635 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.763147    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.765635 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.763249    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.765635 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.763284    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.765635 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.763309    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.765635 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.763334    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.765635 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.763361    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-web-config\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.765635 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.763388    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.765635 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.763414    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.765635 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.763441    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.765635 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.763466    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.765635 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.763500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvbqg\" (UniqueName: \"kubernetes.io/projected/bc8df723-b571-4680-b235-ab8b02aa0b7a-kube-api-access-gvbqg\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.765635 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.763530    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc8df723-b571-4680-b235-ab8b02aa0b7a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.765635 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.763588    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.765635 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.763634    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.766477 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.764425    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.766477 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.764949    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.766477 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.765797    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-config\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.766911 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.766880    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc8df723-b571-4680-b235-ab8b02aa0b7a-config-out\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.767484 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.767458    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.767796 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.767771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.768473 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.768447    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.768853 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.768816    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.769697 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.769181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.769697 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.769407    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.771018 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.770990    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.771895 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.771660    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.771895 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.771749    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.772056 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.772034    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc8df723-b571-4680-b235-ab8b02aa0b7a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.772458 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.772411    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.772702 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.772586    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-web-config\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.774211 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.774186    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.776984 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.776958    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.777897 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.777867    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvbqg\" (UniqueName: \"kubernetes.io/projected/bc8df723-b571-4680-b235-ab8b02aa0b7a-kube-api-access-gvbqg\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.778259 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.778237    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.880977 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.880888    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:46.965908 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.965867    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs\") pod \"network-metrics-daemon-4rggc\" (UID: \"17b91ece-5833-48f6-95f1-7df776a892f8\") " pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:41:46.968814 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.968779    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b91ece-5833-48f6-95f1-7df776a892f8-metrics-certs\") pod \"network-metrics-daemon-4rggc\" (UID: \"17b91ece-5833-48f6-95f1-7df776a892f8\") " pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:41:46.996186 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:46.996156    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hzzqm\""
Jul 10 18:41:47.004513 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:47.004484    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rggc"
Jul 10 18:41:47.287131 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:47.287042    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4rggc"]
Jul 10 18:41:47.325117 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:47.324970    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:41:47.381668 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:41:47.381627    2561 helpers.go:245] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebef81d2_c0d1_4894_9ca0_391f361b659b.slice/crio-conmon-4065f4f0839849205af733307740a1707d6b8a4a5aa5594bae65f47b9fc4f1c4.scope/cpuset.cpus.effective": read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebef81d2_c0d1_4894_9ca0_391f361b659b.slice/crio-conmon-4065f4f0839849205af733307740a1707d6b8a4a5aa5594bae65f47b9fc4f1c4.scope/cpuset.cpus.effective: no such device
Jul 10 18:41:47.405811 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:41:47.405780    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebef81d2_c0d1_4894_9ca0_391f361b659b.slice/crio-4065f4f0839849205af733307740a1707d6b8a4a5aa5594bae65f47b9fc4f1c4.scope\": RecentStats: unable to find data in memory cache]"
Jul 10 18:41:47.423688 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:41:47.423644    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b91ece_5833_48f6_95f1_7df776a892f8.slice/crio-dea0e7fabda7a521d58790de07c60caaed8d04d6e46b388df9c1c7cb6e889439 WatchSource:0}: Error finding container dea0e7fabda7a521d58790de07c60caaed8d04d6e46b388df9c1c7cb6e889439: Status 404 returned error can't find the container with id dea0e7fabda7a521d58790de07c60caaed8d04d6e46b388df9c1c7cb6e889439
Jul 10 18:41:47.424054 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:41:47.424027    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8df723_b571_4680_b235_ab8b02aa0b7a.slice/crio-0d02dda4fac843479a5bd3fd1abb06c1d938f331bbeec225398bfaa134673071 WatchSource:0}: Error finding container 0d02dda4fac843479a5bd3fd1abb06c1d938f331bbeec225398bfaa134673071: Status 404 returned error can't find the container with id 0d02dda4fac843479a5bd3fd1abb06c1d938f331bbeec225398bfaa134673071
Jul 10 18:41:47.604195 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:47.604160    2561 generic.go:356] "Generic (PLEG): container finished" podID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerID="512b3175a67f02d44d4a132eea06bf1be85582340f989f51e847f5be8b815741" exitCode=0
Jul 10 18:41:47.604652 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:47.604254    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b48ffef0-0728-4741-a349-a28e00ca1d0b","Type":"ContainerDied","Data":"512b3175a67f02d44d4a132eea06bf1be85582340f989f51e847f5be8b815741"}
Jul 10 18:41:47.606390 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:47.606354    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8df723-b571-4680-b235-ab8b02aa0b7a","Type":"ContainerStarted","Data":"0d02dda4fac843479a5bd3fd1abb06c1d938f331bbeec225398bfaa134673071"}
Jul 10 18:41:47.608728 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:47.608537    2561 generic.go:356] "Generic (PLEG): container finished" podID="ebef81d2-c0d1-4894-9ca0-391f361b659b" containerID="4065f4f0839849205af733307740a1707d6b8a4a5aa5594bae65f47b9fc4f1c4" exitCode=0
Jul 10 18:41:47.608728 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:47.608661    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f47gv" event={"ID":"ebef81d2-c0d1-4894-9ca0-391f361b659b","Type":"ContainerDied","Data":"4065f4f0839849205af733307740a1707d6b8a4a5aa5594bae65f47b9fc4f1c4"}
Jul 10 18:41:47.610139 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:47.610041    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4rggc" event={"ID":"17b91ece-5833-48f6-95f1-7df776a892f8","Type":"ContainerStarted","Data":"dea0e7fabda7a521d58790de07c60caaed8d04d6e46b388df9c1c7cb6e889439"}
Jul 10 18:41:47.612214 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:47.612193    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dbkh5" event={"ID":"68b929ae-2f98-46da-b893-82064267e801","Type":"ContainerStarted","Data":"b3b87ce442c622551b1f54aff09eb682ca9b13a215dc9a25f62cb19f876639ea"}
Jul 10 18:41:48.621595 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:48.621504    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f47gv" event={"ID":"ebef81d2-c0d1-4894-9ca0-391f361b659b","Type":"ContainerStarted","Data":"b1ec278c2faec52ebbb89b2c542f9a7b291046adb1af1be0849c739ca015db8d"}
Jul 10 18:41:48.622166 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:48.621621    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f47gv" event={"ID":"ebef81d2-c0d1-4894-9ca0-391f361b659b","Type":"ContainerStarted","Data":"0ad27ba7d3c9014422efd5add6141055a638752e7c221797579058e7e94e16d5"}
Jul 10 18:41:48.625823 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:48.625549    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6" event={"ID":"66182fd1-a634-4db8-b35e-d64b8632eadc","Type":"ContainerStarted","Data":"9e56d39a57c063ae029df0b27126adeadf43b5a648388c78e0d2d64d3b40f623"}
Jul 10 18:41:48.625823 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:48.625641    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6" event={"ID":"66182fd1-a634-4db8-b35e-d64b8632eadc","Type":"ContainerStarted","Data":"30736526d048c30d40e7a704130c7f366d0c705a8c49463e98548815a1053843"}
Jul 10 18:41:48.625823 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:48.625681    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6" event={"ID":"66182fd1-a634-4db8-b35e-d64b8632eadc","Type":"ContainerStarted","Data":"675e5181b9973c905bca986301389129ab6a3041a77ed036f89b9315109c2162"}
Jul 10 18:41:48.628600 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:48.628568    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp" event={"ID":"1fb2c50d-7009-417b-927b-bbb8e45c94e0","Type":"ContainerStarted","Data":"33c748ce939f47117f30f341a5ddb867055ff1a3c904cee0ec70dc9b9a5f2fdd"}
Jul 10 18:41:48.631283 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:48.631252    2561 generic.go:356] "Generic (PLEG): container finished" podID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerID="2048337224e2835a367ebab139c276edc99fe70c71506b72b3637beac3516a10" exitCode=0
Jul 10 18:41:48.631393 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:48.631304    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8df723-b571-4680-b235-ab8b02aa0b7a","Type":"ContainerDied","Data":"2048337224e2835a367ebab139c276edc99fe70c71506b72b3637beac3516a10"}
Jul 10 18:41:48.643884 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:48.643813    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-f47gv" podStartSLOduration=5.050731551 podStartE2EDuration="8.64379245s" totalImagesPullingTime="3.502875957s" totalInitContainerRuntime="90.184942ms" isStatefulPod=true podCreationTimestamp="2026-07-10 18:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:48.642687158 +0000 UTC m=+132.116780047" watchObservedRunningTime="2026-07-10 18:41:48.64379245 +0000 UTC m=+132.117885335"
Jul 10 18:41:48.664451 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:48.664387    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-d9kxp" podStartSLOduration=4.877402231 podStartE2EDuration="8.664362598s" totalImagesPullingTime="3.786960367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:48.661831706 +0000 UTC m=+132.135924615" watchObservedRunningTime="2026-07-10 18:41:48.664362598 +0000 UTC m=+132.138455485"
Jul 10 18:41:51.518012 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:51.517980    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:51.580675 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:51.580638    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:52.321272 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:52.321222    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:52.377666 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:52.377633    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:52.652530 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:52.652442    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4rggc" event={"ID":"17b91ece-5833-48f6-95f1-7df776a892f8","Type":"ContainerStarted","Data":"ba9914446cb226024195dc8d9d8f6932ade6a4137daa7f1e5b832305113e0b40"}
Jul 10 18:41:52.654820 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:52.654785    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dbkh5" event={"ID":"68b929ae-2f98-46da-b893-82064267e801","Type":"ContainerStarted","Data":"2ecb66d7a3ca504b17cfa19ff5f20c9cf94a1a8a478ce0f71aba24b4b3603fd4"}
Jul 10 18:41:52.657381 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:52.657355    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6" event={"ID":"66182fd1-a634-4db8-b35e-d64b8632eadc","Type":"ContainerStarted","Data":"3f9a3fe810a0957136b3864354e00414083bfaece466117cf1de0841e8416921"}
Jul 10 18:41:52.659593 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:52.659571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b48ffef0-0728-4741-a349-a28e00ca1d0b","Type":"ContainerStarted","Data":"c09cfbfd807249321635779b37285531058933aead2a8265f2afbbde07679768"}
Jul 10 18:41:52.676203 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:52.676153    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dbkh5" podStartSLOduration=40.889261252 podStartE2EDuration="53.676135438s" totalImagesPullingTime="12.786874186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:52.675259359 +0000 UTC m=+136.149352245" watchObservedRunningTime="2026-07-10 18:41:52.676135438 +0000 UTC m=+136.150228324"
Jul 10 18:41:53.666199 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:53.666173    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4rggc" event={"ID":"17b91ece-5833-48f6-95f1-7df776a892f8","Type":"ContainerStarted","Data":"3641a635af6c35ae04d1ac678b989bc44f521f9579fad1d077f459451011626b"}
Jul 10 18:41:53.680167 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:53.680124    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6" event={"ID":"66182fd1-a634-4db8-b35e-d64b8632eadc","Type":"ContainerStarted","Data":"ccfac8581ea1dc03ee4d4766741a70a12c3bc375fc38c13f35621cdae1ab8194"}
Jul 10 18:41:53.680350 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:53.680213    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6" event={"ID":"66182fd1-a634-4db8-b35e-d64b8632eadc","Type":"ContainerStarted","Data":"c25de5939ef9d0c155706709c642f07ad1186b0c39934ef7ec39e8dd1fc265fa"}
Jul 10 18:41:53.683283 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:53.683259    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:53.693665 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:53.693599    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b48ffef0-0728-4741-a349-a28e00ca1d0b","Type":"ContainerStarted","Data":"836d6f9d045f56bb1d268a05af010efcf6d0ed8118a35ac791f2158df37aed06"}
Jul 10 18:41:53.693780 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:53.693679    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b48ffef0-0728-4741-a349-a28e00ca1d0b","Type":"ContainerStarted","Data":"8a30aba07e9b2fc2bc0f67e0c17bbf5d179a3f9858766af57c9b7fbc52359166"}
Jul 10 18:41:53.693780 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:53.693722    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b48ffef0-0728-4741-a349-a28e00ca1d0b","Type":"ContainerStarted","Data":"8ace2fd4cbad9aad200739bdd05ea49d95ec463794419b86c0e8b9d20f625284"}
Jul 10 18:41:53.693780 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:53.693759    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b48ffef0-0728-4741-a349-a28e00ca1d0b","Type":"ContainerStarted","Data":"63fbddfd8815bbd2b683a496b0872c2d28d1529cf7c3ebf00887f83de094a8fb"}
Jul 10 18:41:53.693991 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:53.693814    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4rggc" podStartSLOduration=131.527077344 podStartE2EDuration="2m15.693799585s" totalImagesPullingTime="4.166722241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:53.693242691 +0000 UTC m=+137.167335605" watchObservedRunningTime="2026-07-10 18:41:53.693799585 +0000 UTC m=+137.167892472"
Jul 10 18:41:53.701942 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:53.701899    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8df723-b571-4680-b235-ab8b02aa0b7a","Type":"ContainerStarted","Data":"347f89fee1940927f3822eb130dc6ec98c9ed1bfb3c891ae736c6129dfe8b99b"}
Jul 10 18:41:53.702198 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:53.702178    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8df723-b571-4680-b235-ab8b02aa0b7a","Type":"ContainerStarted","Data":"bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3"}
Jul 10 18:41:53.705273 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:53.705248    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6"
Jul 10 18:41:53.719367 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:53.719308    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5dd7ccc499-69tw6" podStartSLOduration=3.704596029 podStartE2EDuration="10.719287535s" totalImagesPullingTime="7.014691506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:53.718603289 +0000 UTC m=+137.192696174" watchObservedRunningTime="2026-07-10 18:41:53.719287535 +0000 UTC m=+137.193380421"
Jul 10 18:41:53.793421 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:53.793390    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:53.893024 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:53.892989    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:54.400844 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:54.400756    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:54.500498 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:54.500455    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:54.710187 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:54.710146    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b48ffef0-0728-4741-a349-a28e00ca1d0b","Type":"ContainerStarted","Data":"2a7b1f195c8075c26170dab777692dc993c273f85937dc76c774aaa3873d7ce1"}
Jul 10 18:41:54.713483 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:54.713448    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8df723-b571-4680-b235-ab8b02aa0b7a","Type":"ContainerStarted","Data":"8c2d8e3dce1475802755f3589aa8c166f866de620b30cf298fb11ce43ce1b740"}
Jul 10 18:41:54.713641 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:54.713510    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8df723-b571-4680-b235-ab8b02aa0b7a","Type":"ContainerStarted","Data":"b6e6839f22b6e7a6eebe1a0eadb8a3e249728f34d7bec102ce947aae167fdacc"}
Jul 10 18:41:54.713641 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:54.713535    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8df723-b571-4680-b235-ab8b02aa0b7a","Type":"ContainerStarted","Data":"68b1314deb3ca3e9750582ec3f030d11294f0e75f52d09bce05940a360bd119b"}
Jul 10 18:41:54.713641 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:54.713559    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8df723-b571-4680-b235-ab8b02aa0b7a","Type":"ContainerStarted","Data":"f197c15bc6881922f6b36fd50d1b024e82be25a53fc4ecfa80ec88412142ab12"}
Jul 10 18:41:54.743397 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:54.743327    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.654619219 podStartE2EDuration="13.74330676s" totalImagesPullingTime="7.07333722s" totalInitContainerRuntime="15.350321ms" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:54.742336651 +0000 UTC m=+138.216429598" watchObservedRunningTime="2026-07-10 18:41:54.74330676 +0000 UTC m=+138.217399647"
Jul 10 18:41:54.776244 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:54.776172    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.060853824 podStartE2EDuration="8.776145131s" totalImagesPullingTime="4.694238819s" totalInitContainerRuntime="21.052488ms" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:54.774107542 +0000 UTC m=+138.248200452" watchObservedRunningTime="2026-07-10 18:41:54.776145131 +0000 UTC m=+138.250238018"
Jul 10 18:41:55.521116 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:55.521056    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:55.620719 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:55.620680    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:56.147007 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:56.146968    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:56.211811 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:56.211781    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:56.881506 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:56.881470    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:57.970100 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:57.967372    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:58.065567 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:58.065527    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:58.172289 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:58.172260    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:58.964407 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:58.964361    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:59.068577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:41:59.068539    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:06.728598 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:06.728543    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-d84bdf484-l56g2" podUID="bd2816d5-3984-4084-9706-111c92941084" containerName="registry" containerID="cri-o://9a28f9141b34959b3ad678cb2ca33ce906c984aa29dc132c81acea9988f2c1ff" gracePeriod=30
Jul 10 18:42:07.516050 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.516021    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:42:07.548399 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.548367    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd2816d5-3984-4084-9706-111c92941084-installation-pull-secrets\") pod \"bd2816d5-3984-4084-9706-111c92941084\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") "
Jul 10 18:42:07.548596 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.548427    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-bound-sa-token\") pod \"bd2816d5-3984-4084-9706-111c92941084\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") "
Jul 10 18:42:07.548596 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.548475    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls\") pod \"bd2816d5-3984-4084-9706-111c92941084\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") "
Jul 10 18:42:07.548596 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.548511    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd2816d5-3984-4084-9706-111c92941084-registry-certificates\") pod \"bd2816d5-3984-4084-9706-111c92941084\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") "
Jul 10 18:42:07.548596 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.548529    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd2816d5-3984-4084-9706-111c92941084-trusted-ca\") pod \"bd2816d5-3984-4084-9706-111c92941084\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") "
Jul 10 18:42:07.548596 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.548559    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2d4qb\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-kube-api-access-2d4qb\") pod \"bd2816d5-3984-4084-9706-111c92941084\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") "
Jul 10 18:42:07.548873 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.548619    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd2816d5-3984-4084-9706-111c92941084-ca-trust-extracted\") pod \"bd2816d5-3984-4084-9706-111c92941084\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") "
Jul 10 18:42:07.548873 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.548662    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd2816d5-3984-4084-9706-111c92941084-image-registry-private-configuration\") pod \"bd2816d5-3984-4084-9706-111c92941084\" (UID: \"bd2816d5-3984-4084-9706-111c92941084\") "
Jul 10 18:42:07.549012 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.548981    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2816d5-3984-4084-9706-111c92941084-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bd2816d5-3984-4084-9706-111c92941084" (UID: "bd2816d5-3984-4084-9706-111c92941084"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:42:07.549355 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.549324    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2816d5-3984-4084-9706-111c92941084-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bd2816d5-3984-4084-9706-111c92941084" (UID: "bd2816d5-3984-4084-9706-111c92941084"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:42:07.552162 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.552106    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bd2816d5-3984-4084-9706-111c92941084" (UID: "bd2816d5-3984-4084-9706-111c92941084"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:42:07.552579 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.552515    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bd2816d5-3984-4084-9706-111c92941084" (UID: "bd2816d5-3984-4084-9706-111c92941084"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:42:07.552689 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.552522    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-kube-api-access-2d4qb" (OuterVolumeSpecName: "kube-api-access-2d4qb") pod "bd2816d5-3984-4084-9706-111c92941084" (UID: "bd2816d5-3984-4084-9706-111c92941084"). InnerVolumeSpecName "kube-api-access-2d4qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:42:07.552689 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.552528    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd2816d5-3984-4084-9706-111c92941084-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bd2816d5-3984-4084-9706-111c92941084" (UID: "bd2816d5-3984-4084-9706-111c92941084"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:42:07.552805 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.552785    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd2816d5-3984-4084-9706-111c92941084-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bd2816d5-3984-4084-9706-111c92941084" (UID: "bd2816d5-3984-4084-9706-111c92941084"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:42:07.558347 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.558321    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2816d5-3984-4084-9706-111c92941084-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bd2816d5-3984-4084-9706-111c92941084" (UID: "bd2816d5-3984-4084-9706-111c92941084"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:42:07.650024 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.649934    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd2816d5-3984-4084-9706-111c92941084-image-registry-private-configuration\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:42:07.650024 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.649969    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd2816d5-3984-4084-9706-111c92941084-installation-pull-secrets\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:42:07.650024 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.649980    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-bound-sa-token\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:42:07.650024 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.649989    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-registry-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:42:07.650024 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.649997    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd2816d5-3984-4084-9706-111c92941084-registry-certificates\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:42:07.650024 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.650007    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd2816d5-3984-4084-9706-111c92941084-trusted-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:42:07.650024 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.650015    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d4qb\" (UniqueName: \"kubernetes.io/projected/bd2816d5-3984-4084-9706-111c92941084-kube-api-access-2d4qb\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:42:07.650024 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.650023    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd2816d5-3984-4084-9706-111c92941084-ca-trust-extracted\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:42:07.768731 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.768688    2561 generic.go:356] "Generic (PLEG): container finished" podID="bd2816d5-3984-4084-9706-111c92941084" containerID="9a28f9141b34959b3ad678cb2ca33ce906c984aa29dc132c81acea9988f2c1ff" exitCode=0
Jul 10 18:42:07.768731 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.768731    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d84bdf484-l56g2"
Jul 10 18:42:07.769262 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.768771    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d84bdf484-l56g2" event={"ID":"bd2816d5-3984-4084-9706-111c92941084","Type":"ContainerDied","Data":"9a28f9141b34959b3ad678cb2ca33ce906c984aa29dc132c81acea9988f2c1ff"}
Jul 10 18:42:07.769262 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.768843    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d84bdf484-l56g2" event={"ID":"bd2816d5-3984-4084-9706-111c92941084","Type":"ContainerDied","Data":"ebf04fab9acc7afb2abe53123da9c4abe852be4053059f0971182c71f3e37352"}
Jul 10 18:42:07.769262 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.768863    2561 scope.go:117] "RemoveContainer" containerID="9a28f9141b34959b3ad678cb2ca33ce906c984aa29dc132c81acea9988f2c1ff"
Jul 10 18:42:07.778146 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.778127    2561 scope.go:117] "RemoveContainer" containerID="9a28f9141b34959b3ad678cb2ca33ce906c984aa29dc132c81acea9988f2c1ff"
Jul 10 18:42:07.778436 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:42:07.778415    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a28f9141b34959b3ad678cb2ca33ce906c984aa29dc132c81acea9988f2c1ff\": container with ID starting with 9a28f9141b34959b3ad678cb2ca33ce906c984aa29dc132c81acea9988f2c1ff not found: ID does not exist" containerID="9a28f9141b34959b3ad678cb2ca33ce906c984aa29dc132c81acea9988f2c1ff"
Jul 10 18:42:07.778491 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.778446    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a28f9141b34959b3ad678cb2ca33ce906c984aa29dc132c81acea9988f2c1ff"} err="failed to get container status \"9a28f9141b34959b3ad678cb2ca33ce906c984aa29dc132c81acea9988f2c1ff\": rpc error: code = NotFound desc = could not find container \"9a28f9141b34959b3ad678cb2ca33ce906c984aa29dc132c81acea9988f2c1ff\": container with ID starting with 9a28f9141b34959b3ad678cb2ca33ce906c984aa29dc132c81acea9988f2c1ff not found: ID does not exist"
Jul 10 18:42:07.791410 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.791312    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d84bdf484-l56g2"]
Jul 10 18:42:07.793225 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:07.793199    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d84bdf484-l56g2"]
Jul 10 18:42:09.071884 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:09.071850    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd2816d5-3984-4084-9706-111c92941084" path="/var/lib/kubelet/pods/bd2816d5-3984-4084-9706-111c92941084/volumes"
Jul 10 18:42:09.398824 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:09.398734    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:09.475482 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:09.475442    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:09.728983 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:09.728950    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:09.829129 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:09.829096    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:09.930511 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:09.930474    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:12.791115 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:12.791067    2561 generic.go:356] "Generic (PLEG): container finished" podID="a4adda52-8024-4b86-a7f0-6f6ee4d9e731" containerID="20be0072753401e4fde731f1b25d7b84752240fa42e7dc962da97d08dc828eb5" exitCode=0
Jul 10 18:42:12.791566 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:12.791138    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2" event={"ID":"a4adda52-8024-4b86-a7f0-6f6ee4d9e731","Type":"ContainerDied","Data":"20be0072753401e4fde731f1b25d7b84752240fa42e7dc962da97d08dc828eb5"}
Jul 10 18:42:12.791566 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:12.791481    2561 scope.go:117] "RemoveContainer" containerID="20be0072753401e4fde731f1b25d7b84752240fa42e7dc962da97d08dc828eb5"
Jul 10 18:42:13.797208 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:13.797171    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5j8c2" event={"ID":"a4adda52-8024-4b86-a7f0-6f6ee4d9e731","Type":"ContainerStarted","Data":"5f8041eeb3eb3448108591cfe4b149351ba4f30d6127905a6a1753a65758baf1"}
Jul 10 18:42:22.834951 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:22.834901    2561 generic.go:356] "Generic (PLEG): container finished" podID="27e2c236-1c19-48b0-ad1a-38d02e492b6c" containerID="62e27793672a13b0f3ef83bda59d26ba2d772f979f822337bb66151b021c1074" exitCode=0
Jul 10 18:42:22.835460 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:22.835049    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9" event={"ID":"27e2c236-1c19-48b0-ad1a-38d02e492b6c","Type":"ContainerDied","Data":"62e27793672a13b0f3ef83bda59d26ba2d772f979f822337bb66151b021c1074"}
Jul 10 18:42:22.835554 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:22.835468    2561 scope.go:117] "RemoveContainer" containerID="62e27793672a13b0f3ef83bda59d26ba2d772f979f822337bb66151b021c1074"
Jul 10 18:42:23.840916 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:23.840882    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-v7qt9" event={"ID":"27e2c236-1c19-48b0-ad1a-38d02e492b6c","Type":"ContainerStarted","Data":"185e46c6edb961bf39e416c6877746e71ebd57d4fb3c90fe7979b29be9986d67"}
Jul 10 18:42:24.395560 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:24.395510    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:24.453097 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:24.453052    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:46.881242 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:46.881201    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:46.900695 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:46.900665    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:46.965585 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:46.965558    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:52.972193 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:52.972153    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:59.471849 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:59.471810    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:59.748193 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:42:59.748098    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:00.588425 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:00.588386    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 10 18:43:00.588853 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:00.588673    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="alertmanager" containerID="cri-o://c09cfbfd807249321635779b37285531058933aead2a8265f2afbbde07679768" gracePeriod=120
Jul 10 18:43:00.588853 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:00.588746    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="kube-rbac-proxy-web" containerID="cri-o://8ace2fd4cbad9aad200739bdd05ea49d95ec463794419b86c0e8b9d20f625284" gracePeriod=120
Jul 10 18:43:00.588853 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:00.588739    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="kube-rbac-proxy-metric" containerID="cri-o://836d6f9d045f56bb1d268a05af010efcf6d0ed8118a35ac791f2158df37aed06" gracePeriod=120
Jul 10 18:43:00.588853 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:00.588807    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="config-reloader" containerID="cri-o://63fbddfd8815bbd2b683a496b0872c2d28d1529cf7c3ebf00887f83de094a8fb" gracePeriod=120
Jul 10 18:43:00.588853 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:00.588768    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="kube-rbac-proxy" containerID="cri-o://8a30aba07e9b2fc2bc0f67e0c17bbf5d179a3f9858766af57c9b7fbc52359166" gracePeriod=120
Jul 10 18:43:00.589154 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:00.588853    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="prom-label-proxy" containerID="cri-o://2a7b1f195c8075c26170dab777692dc993c273f85937dc76c774aaa3873d7ce1" gracePeriod=120
Jul 10 18:43:01.008811 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.008771    2561 generic.go:356] "Generic (PLEG): container finished" podID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerID="2a7b1f195c8075c26170dab777692dc993c273f85937dc76c774aaa3873d7ce1" exitCode=0
Jul 10 18:43:01.008811 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.008798    2561 generic.go:356] "Generic (PLEG): container finished" podID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerID="836d6f9d045f56bb1d268a05af010efcf6d0ed8118a35ac791f2158df37aed06" exitCode=0
Jul 10 18:43:01.008811 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.008806    2561 generic.go:356] "Generic (PLEG): container finished" podID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerID="8a30aba07e9b2fc2bc0f67e0c17bbf5d179a3f9858766af57c9b7fbc52359166" exitCode=0
Jul 10 18:43:01.008811 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.008812    2561 generic.go:356] "Generic (PLEG): container finished" podID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerID="63fbddfd8815bbd2b683a496b0872c2d28d1529cf7c3ebf00887f83de094a8fb" exitCode=0
Jul 10 18:43:01.008811 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.008820    2561 generic.go:356] "Generic (PLEG): container finished" podID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerID="c09cfbfd807249321635779b37285531058933aead2a8265f2afbbde07679768" exitCode=0
Jul 10 18:43:01.009126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.008837    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b48ffef0-0728-4741-a349-a28e00ca1d0b","Type":"ContainerDied","Data":"2a7b1f195c8075c26170dab777692dc993c273f85937dc76c774aaa3873d7ce1"}
Jul 10 18:43:01.009126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.008902    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b48ffef0-0728-4741-a349-a28e00ca1d0b","Type":"ContainerDied","Data":"836d6f9d045f56bb1d268a05af010efcf6d0ed8118a35ac791f2158df37aed06"}
Jul 10 18:43:01.009126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.008928    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b48ffef0-0728-4741-a349-a28e00ca1d0b","Type":"ContainerDied","Data":"8a30aba07e9b2fc2bc0f67e0c17bbf5d179a3f9858766af57c9b7fbc52359166"}
Jul 10 18:43:01.009126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.008953    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b48ffef0-0728-4741-a349-a28e00ca1d0b","Type":"ContainerDied","Data":"63fbddfd8815bbd2b683a496b0872c2d28d1529cf7c3ebf00887f83de094a8fb"}
Jul 10 18:43:01.009126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.008975    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b48ffef0-0728-4741-a349-a28e00ca1d0b","Type":"ContainerDied","Data":"c09cfbfd807249321635779b37285531058933aead2a8265f2afbbde07679768"}
Jul 10 18:43:01.846729 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.846700    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:01.934836 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.934801    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b48ffef0-0728-4741-a349-a28e00ca1d0b-config-out\") pod \"b48ffef0-0728-4741-a349-a28e00ca1d0b\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") "
Jul 10 18:43:01.934836 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.934843    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddd24\" (UniqueName: \"kubernetes.io/projected/b48ffef0-0728-4741-a349-a28e00ca1d0b-kube-api-access-ddd24\") pod \"b48ffef0-0728-4741-a349-a28e00ca1d0b\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") "
Jul 10 18:43:01.935070 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.934878    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b48ffef0-0728-4741-a349-a28e00ca1d0b-alertmanager-trusted-ca-bundle\") pod \"b48ffef0-0728-4741-a349-a28e00ca1d0b\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") "
Jul 10 18:43:01.935070 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.934917    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b48ffef0-0728-4741-a349-a28e00ca1d0b-metrics-client-ca\") pod \"b48ffef0-0728-4741-a349-a28e00ca1d0b\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") "
Jul 10 18:43:01.935070 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.934935    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-kube-rbac-proxy-web\") pod \"b48ffef0-0728-4741-a349-a28e00ca1d0b\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") "
Jul 10 18:43:01.935070 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.934969    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-web-config\") pod \"b48ffef0-0728-4741-a349-a28e00ca1d0b\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") "
Jul 10 18:43:01.935070 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.934999    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-kube-rbac-proxy\") pod \"b48ffef0-0728-4741-a349-a28e00ca1d0b\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") "
Jul 10 18:43:01.935070 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.935030    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-config-volume\") pod \"b48ffef0-0728-4741-a349-a28e00ca1d0b\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") "
Jul 10 18:43:01.935381 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.935091    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b48ffef0-0728-4741-a349-a28e00ca1d0b-tls-assets\") pod \"b48ffef0-0728-4741-a349-a28e00ca1d0b\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") "
Jul 10 18:43:01.935381 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.935118    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b48ffef0-0728-4741-a349-a28e00ca1d0b-alertmanager-main-db\") pod \"b48ffef0-0728-4741-a349-a28e00ca1d0b\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") "
Jul 10 18:43:01.935381 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.935171    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-cluster-tls-config\") pod \"b48ffef0-0728-4741-a349-a28e00ca1d0b\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") "
Jul 10 18:43:01.935381 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.935202    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b48ffef0-0728-4741-a349-a28e00ca1d0b\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") "
Jul 10 18:43:01.935381 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.935243    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-main-tls\") pod \"b48ffef0-0728-4741-a349-a28e00ca1d0b\" (UID: \"b48ffef0-0728-4741-a349-a28e00ca1d0b\") "
Jul 10 18:43:01.935634 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.935599    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48ffef0-0728-4741-a349-a28e00ca1d0b-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "b48ffef0-0728-4741-a349-a28e00ca1d0b" (UID: "b48ffef0-0728-4741-a349-a28e00ca1d0b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:43:01.935634 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.935603    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48ffef0-0728-4741-a349-a28e00ca1d0b-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "b48ffef0-0728-4741-a349-a28e00ca1d0b" (UID: "b48ffef0-0728-4741-a349-a28e00ca1d0b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:43:01.936548 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.936513    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48ffef0-0728-4741-a349-a28e00ca1d0b-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "b48ffef0-0728-4741-a349-a28e00ca1d0b" (UID: "b48ffef0-0728-4741-a349-a28e00ca1d0b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:43:01.938471 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.938399    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "b48ffef0-0728-4741-a349-a28e00ca1d0b" (UID: "b48ffef0-0728-4741-a349-a28e00ca1d0b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:43:01.939127 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.939069    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48ffef0-0728-4741-a349-a28e00ca1d0b-config-out" (OuterVolumeSpecName: "config-out") pod "b48ffef0-0728-4741-a349-a28e00ca1d0b" (UID: "b48ffef0-0728-4741-a349-a28e00ca1d0b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:43:01.939332 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.939271    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "b48ffef0-0728-4741-a349-a28e00ca1d0b" (UID: "b48ffef0-0728-4741-a349-a28e00ca1d0b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:43:01.939332 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.939276    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48ffef0-0728-4741-a349-a28e00ca1d0b-kube-api-access-ddd24" (OuterVolumeSpecName: "kube-api-access-ddd24") pod "b48ffef0-0728-4741-a349-a28e00ca1d0b" (UID: "b48ffef0-0728-4741-a349-a28e00ca1d0b"). InnerVolumeSpecName "kube-api-access-ddd24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:43:01.939565 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.939527    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-config-volume" (OuterVolumeSpecName: "config-volume") pod "b48ffef0-0728-4741-a349-a28e00ca1d0b" (UID: "b48ffef0-0728-4741-a349-a28e00ca1d0b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:43:01.940433 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.940405    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "b48ffef0-0728-4741-a349-a28e00ca1d0b" (UID: "b48ffef0-0728-4741-a349-a28e00ca1d0b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:43:01.940653 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.940632    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48ffef0-0728-4741-a349-a28e00ca1d0b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b48ffef0-0728-4741-a349-a28e00ca1d0b" (UID: "b48ffef0-0728-4741-a349-a28e00ca1d0b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:43:01.941050 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.941026    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "b48ffef0-0728-4741-a349-a28e00ca1d0b" (UID: "b48ffef0-0728-4741-a349-a28e00ca1d0b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:43:01.942946 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.942922    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "b48ffef0-0728-4741-a349-a28e00ca1d0b" (UID: "b48ffef0-0728-4741-a349-a28e00ca1d0b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:43:01.950861 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:01.950821    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-web-config" (OuterVolumeSpecName: "web-config") pod "b48ffef0-0728-4741-a349-a28e00ca1d0b" (UID: "b48ffef0-0728-4741-a349-a28e00ca1d0b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:43:02.016688 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.016651    2561 generic.go:356] "Generic (PLEG): container finished" podID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerID="8ace2fd4cbad9aad200739bdd05ea49d95ec463794419b86c0e8b9d20f625284" exitCode=0
Jul 10 18:43:02.016888 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.016729    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b48ffef0-0728-4741-a349-a28e00ca1d0b","Type":"ContainerDied","Data":"8ace2fd4cbad9aad200739bdd05ea49d95ec463794419b86c0e8b9d20f625284"}
Jul 10 18:43:02.016888 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.016752    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.016888 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.016818    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b48ffef0-0728-4741-a349-a28e00ca1d0b","Type":"ContainerDied","Data":"969d2ec698b2ba4cb1f2e6f0dbcfeb15e6f74615b9801549c730abbebc19af4c"}
Jul 10 18:43:02.016888 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.016867    2561 scope.go:117] "RemoveContainer" containerID="2a7b1f195c8075c26170dab777692dc993c273f85937dc76c774aaa3873d7ce1"
Jul 10 18:43:02.025622 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.025604    2561 scope.go:117] "RemoveContainer" containerID="836d6f9d045f56bb1d268a05af010efcf6d0ed8118a35ac791f2158df37aed06"
Jul 10 18:43:02.033504 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.033478    2561 scope.go:117] "RemoveContainer" containerID="8a30aba07e9b2fc2bc0f67e0c17bbf5d179a3f9858766af57c9b7fbc52359166"
Jul 10 18:43:02.036867 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.036840    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b48ffef0-0728-4741-a349-a28e00ca1d0b-tls-assets\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:02.036867 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.036869    2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b48ffef0-0728-4741-a349-a28e00ca1d0b-alertmanager-main-db\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:02.037051 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.036883    2561 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-cluster-tls-config\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:02.037051 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.036897    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:02.037051 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.036909    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-main-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:02.037051 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.036921    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b48ffef0-0728-4741-a349-a28e00ca1d0b-config-out\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:02.037051 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.036934    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddd24\" (UniqueName: \"kubernetes.io/projected/b48ffef0-0728-4741-a349-a28e00ca1d0b-kube-api-access-ddd24\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:02.037051 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.036948    2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b48ffef0-0728-4741-a349-a28e00ca1d0b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:02.037051 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.036960    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b48ffef0-0728-4741-a349-a28e00ca1d0b-metrics-client-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:02.037051 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.036975    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:02.037051 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.036987    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-web-config\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:02.037051 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.036999    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:02.037051 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.037012    2561 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b48ffef0-0728-4741-a349-a28e00ca1d0b-config-volume\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:02.041229 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.041208    2561 scope.go:117] "RemoveContainer" containerID="8ace2fd4cbad9aad200739bdd05ea49d95ec463794419b86c0e8b9d20f625284"
Jul 10 18:43:02.045642 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.045609    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 10 18:43:02.050262 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.050238    2561 scope.go:117] "RemoveContainer" containerID="63fbddfd8815bbd2b683a496b0872c2d28d1529cf7c3ebf00887f83de094a8fb"
Jul 10 18:43:02.058097 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.058063    2561 scope.go:117] "RemoveContainer" containerID="c09cfbfd807249321635779b37285531058933aead2a8265f2afbbde07679768"
Jul 10 18:43:02.060257 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.060233    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 10 18:43:02.066126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.066064    2561 scope.go:117] "RemoveContainer" containerID="512b3175a67f02d44d4a132eea06bf1be85582340f989f51e847f5be8b815741"
Jul 10 18:43:02.073932 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.073909    2561 scope.go:117] "RemoveContainer" containerID="2a7b1f195c8075c26170dab777692dc993c273f85937dc76c774aaa3873d7ce1"
Jul 10 18:43:02.074330 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:02.074306    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a7b1f195c8075c26170dab777692dc993c273f85937dc76c774aaa3873d7ce1\": container with ID starting with 2a7b1f195c8075c26170dab777692dc993c273f85937dc76c774aaa3873d7ce1 not found: ID does not exist" containerID="2a7b1f195c8075c26170dab777692dc993c273f85937dc76c774aaa3873d7ce1"
Jul 10 18:43:02.074392 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.074342    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a7b1f195c8075c26170dab777692dc993c273f85937dc76c774aaa3873d7ce1"} err="failed to get container status \"2a7b1f195c8075c26170dab777692dc993c273f85937dc76c774aaa3873d7ce1\": rpc error: code = NotFound desc = could not find container \"2a7b1f195c8075c26170dab777692dc993c273f85937dc76c774aaa3873d7ce1\": container with ID starting with 2a7b1f195c8075c26170dab777692dc993c273f85937dc76c774aaa3873d7ce1 not found: ID does not exist"
Jul 10 18:43:02.074392 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.074365    2561 scope.go:117] "RemoveContainer" containerID="836d6f9d045f56bb1d268a05af010efcf6d0ed8118a35ac791f2158df37aed06"
Jul 10 18:43:02.074658 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:02.074636    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836d6f9d045f56bb1d268a05af010efcf6d0ed8118a35ac791f2158df37aed06\": container with ID starting with 836d6f9d045f56bb1d268a05af010efcf6d0ed8118a35ac791f2158df37aed06 not found: ID does not exist" containerID="836d6f9d045f56bb1d268a05af010efcf6d0ed8118a35ac791f2158df37aed06"
Jul 10 18:43:02.074725 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.074669    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836d6f9d045f56bb1d268a05af010efcf6d0ed8118a35ac791f2158df37aed06"} err="failed to get container status \"836d6f9d045f56bb1d268a05af010efcf6d0ed8118a35ac791f2158df37aed06\": rpc error: code = NotFound desc = could not find container \"836d6f9d045f56bb1d268a05af010efcf6d0ed8118a35ac791f2158df37aed06\": container with ID starting with 836d6f9d045f56bb1d268a05af010efcf6d0ed8118a35ac791f2158df37aed06 not found: ID does not exist"
Jul 10 18:43:02.074725 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.074694    2561 scope.go:117] "RemoveContainer" containerID="8a30aba07e9b2fc2bc0f67e0c17bbf5d179a3f9858766af57c9b7fbc52359166"
Jul 10 18:43:02.074937 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:02.074919    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a30aba07e9b2fc2bc0f67e0c17bbf5d179a3f9858766af57c9b7fbc52359166\": container with ID starting with 8a30aba07e9b2fc2bc0f67e0c17bbf5d179a3f9858766af57c9b7fbc52359166 not found: ID does not exist" containerID="8a30aba07e9b2fc2bc0f67e0c17bbf5d179a3f9858766af57c9b7fbc52359166"
Jul 10 18:43:02.074991 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.074941    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a30aba07e9b2fc2bc0f67e0c17bbf5d179a3f9858766af57c9b7fbc52359166"} err="failed to get container status \"8a30aba07e9b2fc2bc0f67e0c17bbf5d179a3f9858766af57c9b7fbc52359166\": rpc error: code = NotFound desc = could not find container \"8a30aba07e9b2fc2bc0f67e0c17bbf5d179a3f9858766af57c9b7fbc52359166\": container with ID starting with 8a30aba07e9b2fc2bc0f67e0c17bbf5d179a3f9858766af57c9b7fbc52359166 not found: ID does not exist"
Jul 10 18:43:02.074991 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.074959    2561 scope.go:117] "RemoveContainer" containerID="8ace2fd4cbad9aad200739bdd05ea49d95ec463794419b86c0e8b9d20f625284"
Jul 10 18:43:02.075225 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:02.075205    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ace2fd4cbad9aad200739bdd05ea49d95ec463794419b86c0e8b9d20f625284\": container with ID starting with 8ace2fd4cbad9aad200739bdd05ea49d95ec463794419b86c0e8b9d20f625284 not found: ID does not exist" containerID="8ace2fd4cbad9aad200739bdd05ea49d95ec463794419b86c0e8b9d20f625284"
Jul 10 18:43:02.075281 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.075232    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ace2fd4cbad9aad200739bdd05ea49d95ec463794419b86c0e8b9d20f625284"} err="failed to get container status \"8ace2fd4cbad9aad200739bdd05ea49d95ec463794419b86c0e8b9d20f625284\": rpc error: code = NotFound desc = could not find container \"8ace2fd4cbad9aad200739bdd05ea49d95ec463794419b86c0e8b9d20f625284\": container with ID starting with 8ace2fd4cbad9aad200739bdd05ea49d95ec463794419b86c0e8b9d20f625284 not found: ID does not exist"
Jul 10 18:43:02.075281 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.075247    2561 scope.go:117] "RemoveContainer" containerID="63fbddfd8815bbd2b683a496b0872c2d28d1529cf7c3ebf00887f83de094a8fb"
Jul 10 18:43:02.075496 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:02.075479    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63fbddfd8815bbd2b683a496b0872c2d28d1529cf7c3ebf00887f83de094a8fb\": container with ID starting with 63fbddfd8815bbd2b683a496b0872c2d28d1529cf7c3ebf00887f83de094a8fb not found: ID does not exist" containerID="63fbddfd8815bbd2b683a496b0872c2d28d1529cf7c3ebf00887f83de094a8fb"
Jul 10 18:43:02.075539 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.075499    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63fbddfd8815bbd2b683a496b0872c2d28d1529cf7c3ebf00887f83de094a8fb"} err="failed to get container status \"63fbddfd8815bbd2b683a496b0872c2d28d1529cf7c3ebf00887f83de094a8fb\": rpc error: code = NotFound desc = could not find container \"63fbddfd8815bbd2b683a496b0872c2d28d1529cf7c3ebf00887f83de094a8fb\": container with ID starting with 63fbddfd8815bbd2b683a496b0872c2d28d1529cf7c3ebf00887f83de094a8fb not found: ID does not exist"
Jul 10 18:43:02.075539 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.075512    2561 scope.go:117] "RemoveContainer" containerID="c09cfbfd807249321635779b37285531058933aead2a8265f2afbbde07679768"
Jul 10 18:43:02.075717 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:02.075703    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c09cfbfd807249321635779b37285531058933aead2a8265f2afbbde07679768\": container with ID starting with c09cfbfd807249321635779b37285531058933aead2a8265f2afbbde07679768 not found: ID does not exist" containerID="c09cfbfd807249321635779b37285531058933aead2a8265f2afbbde07679768"
Jul 10 18:43:02.075761 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.075718    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c09cfbfd807249321635779b37285531058933aead2a8265f2afbbde07679768"} err="failed to get container status \"c09cfbfd807249321635779b37285531058933aead2a8265f2afbbde07679768\": rpc error: code = NotFound desc = could not find container \"c09cfbfd807249321635779b37285531058933aead2a8265f2afbbde07679768\": container with ID starting with c09cfbfd807249321635779b37285531058933aead2a8265f2afbbde07679768 not found: ID does not exist"
Jul 10 18:43:02.075761 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.075729    2561 scope.go:117] "RemoveContainer" containerID="512b3175a67f02d44d4a132eea06bf1be85582340f989f51e847f5be8b815741"
Jul 10 18:43:02.075940 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:02.075926    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"512b3175a67f02d44d4a132eea06bf1be85582340f989f51e847f5be8b815741\": container with ID starting with 512b3175a67f02d44d4a132eea06bf1be85582340f989f51e847f5be8b815741 not found: ID does not exist" containerID="512b3175a67f02d44d4a132eea06bf1be85582340f989f51e847f5be8b815741"
Jul 10 18:43:02.075981 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.075942    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"512b3175a67f02d44d4a132eea06bf1be85582340f989f51e847f5be8b815741"} err="failed to get container status \"512b3175a67f02d44d4a132eea06bf1be85582340f989f51e847f5be8b815741\": rpc error: code = NotFound desc = could not find container \"512b3175a67f02d44d4a132eea06bf1be85582340f989f51e847f5be8b815741\": container with ID starting with 512b3175a67f02d44d4a132eea06bf1be85582340f989f51e847f5be8b815741 not found: ID does not exist"
Jul 10 18:43:02.101507 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101471    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 10 18:43:02.101794 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101781    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="prom-label-proxy"
Jul 10 18:43:02.101834 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101795    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="prom-label-proxy"
Jul 10 18:43:02.101834 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101804    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="init-config-reloader"
Jul 10 18:43:02.101834 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101811    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="init-config-reloader"
Jul 10 18:43:02.101834 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101829    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="kube-rbac-proxy-web"
Jul 10 18:43:02.101948 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101835    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="kube-rbac-proxy-web"
Jul 10 18:43:02.101948 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101846    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd2816d5-3984-4084-9706-111c92941084" containerName="registry"
Jul 10 18:43:02.101948 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101853    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd2816d5-3984-4084-9706-111c92941084" containerName="registry"
Jul 10 18:43:02.101948 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101859    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="alertmanager"
Jul 10 18:43:02.101948 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101864    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="alertmanager"
Jul 10 18:43:02.101948 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101873    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="config-reloader"
Jul 10 18:43:02.101948 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101878    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="config-reloader"
Jul 10 18:43:02.101948 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101884    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="kube-rbac-proxy"
Jul 10 18:43:02.101948 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101889    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="kube-rbac-proxy"
Jul 10 18:43:02.101948 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101898    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="kube-rbac-proxy-metric"
Jul 10 18:43:02.101948 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101903    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="kube-rbac-proxy-metric"
Jul 10 18:43:02.101948 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101949    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="alertmanager"
Jul 10 18:43:02.102297 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101957    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="kube-rbac-proxy-metric"
Jul 10 18:43:02.102297 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101964    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="prom-label-proxy"
Jul 10 18:43:02.102297 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101971    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="kube-rbac-proxy-web"
Jul 10 18:43:02.102297 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101979    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd2816d5-3984-4084-9706-111c92941084" containerName="registry"
Jul 10 18:43:02.102297 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101984    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="kube-rbac-proxy"
Jul 10 18:43:02.102297 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.101990    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" containerName="config-reloader"
Jul 10 18:43:02.106627 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.106601    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.109209 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.109189    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 10 18:43:02.109343 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.109192    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 10 18:43:02.111410 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.111389    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 10 18:43:02.111509 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.111484    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 10 18:43:02.112305 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.112283    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 10 18:43:02.112475 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.112453    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 10 18:43:02.112585 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.112480    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 10 18:43:02.112698 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.112683    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 10 18:43:02.116157 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.116131    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hxmg9\""
Jul 10 18:43:02.121067 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.121042    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 10 18:43:02.135943 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.135912    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 10 18:43:02.138406 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.138377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-config-volume\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.138536 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.138410    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.138536 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.138502    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66a2f044-d6d4-401c-b49e-19f73a1bb034-config-out\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.138604 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.138544    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-web-config\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.138604 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.138580    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.138672 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.138607    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.138672 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.138630    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.138672 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.138648    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crrsw\" (UniqueName: \"kubernetes.io/projected/66a2f044-d6d4-401c-b49e-19f73a1bb034-kube-api-access-crrsw\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.138780 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.138686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66a2f044-d6d4-401c-b49e-19f73a1bb034-tls-assets\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.138780 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.138709    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/66a2f044-d6d4-401c-b49e-19f73a1bb034-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.138780 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.138747    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a2f044-d6d4-401c-b49e-19f73a1bb034-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.138879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.138786    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.138879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.138809    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66a2f044-d6d4-401c-b49e-19f73a1bb034-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.240150 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.240013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.240150 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.240055    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crrsw\" (UniqueName: \"kubernetes.io/projected/66a2f044-d6d4-401c-b49e-19f73a1bb034-kube-api-access-crrsw\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.240150 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.240114    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66a2f044-d6d4-401c-b49e-19f73a1bb034-tls-assets\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.240150 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.240133    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/66a2f044-d6d4-401c-b49e-19f73a1bb034-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.240473 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.240165    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a2f044-d6d4-401c-b49e-19f73a1bb034-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.240473 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.240193    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.240473 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.240216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66a2f044-d6d4-401c-b49e-19f73a1bb034-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.240473 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.240244    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-config-volume\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.240473 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.240273    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.240473 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.240320    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66a2f044-d6d4-401c-b49e-19f73a1bb034-config-out\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.240473 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.240343    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-web-config\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.240473 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.240364    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.240473 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.240400    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.241009 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.240980    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/66a2f044-d6d4-401c-b49e-19f73a1bb034-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.241323 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.241299    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a2f044-d6d4-401c-b49e-19f73a1bb034-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.241899 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.241874    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66a2f044-d6d4-401c-b49e-19f73a1bb034-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.243834 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.243697    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66a2f044-d6d4-401c-b49e-19f73a1bb034-config-out\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.243951 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.243903    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-config-volume\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.244100 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.244034    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.244529 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.244505    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66a2f044-d6d4-401c-b49e-19f73a1bb034-tls-assets\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.244616 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.244566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-web-config\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.244696 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.244671    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.244937 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.244842    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.245046 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.245024    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.245815 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.245782    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66a2f044-d6d4-401c-b49e-19f73a1bb034-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.251147 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.251110    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crrsw\" (UniqueName: \"kubernetes.io/projected/66a2f044-d6d4-401c-b49e-19f73a1bb034-kube-api-access-crrsw\") pod \"alertmanager-main-0\" (UID: \"66a2f044-d6d4-401c-b49e-19f73a1bb034\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.415803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.415764    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 10 18:43:02.569944 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:43:02.569912    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a2f044_d6d4_401c_b49e_19f73a1bb034.slice/crio-415cb4d0167d00d62ed69cca2a5e84c9bbac02f7f37902220a55fa75178722eb WatchSource:0}: Error finding container 415cb4d0167d00d62ed69cca2a5e84c9bbac02f7f37902220a55fa75178722eb: Status 404 returned error can't find the container with id 415cb4d0167d00d62ed69cca2a5e84c9bbac02f7f37902220a55fa75178722eb
Jul 10 18:43:02.570564 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:02.570541    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 10 18:43:03.021901 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:03.021866    2561 generic.go:356] "Generic (PLEG): container finished" podID="66a2f044-d6d4-401c-b49e-19f73a1bb034" containerID="2d6aa18fae0d773128857324392d22dde34f3d950d2ed4014b2ab12e7f17500d" exitCode=0
Jul 10 18:43:03.022338 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:03.021950    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66a2f044-d6d4-401c-b49e-19f73a1bb034","Type":"ContainerDied","Data":"2d6aa18fae0d773128857324392d22dde34f3d950d2ed4014b2ab12e7f17500d"}
Jul 10 18:43:03.022338 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:03.022036    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66a2f044-d6d4-401c-b49e-19f73a1bb034","Type":"ContainerStarted","Data":"415cb4d0167d00d62ed69cca2a5e84c9bbac02f7f37902220a55fa75178722eb"}
Jul 10 18:43:03.072714 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:03.072686    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b48ffef0-0728-4741-a349-a28e00ca1d0b" path="/var/lib/kubelet/pods/b48ffef0-0728-4741-a349-a28e00ca1d0b/volumes"
Jul 10 18:43:04.030700 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:04.030663    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66a2f044-d6d4-401c-b49e-19f73a1bb034","Type":"ContainerStarted","Data":"4a1eb2450c6f31d6e0872fce768e9d077bf425136f2807658370eb30215864a3"}
Jul 10 18:43:04.031143 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:04.030746    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66a2f044-d6d4-401c-b49e-19f73a1bb034","Type":"ContainerStarted","Data":"0b536ae26128fc6d6290971bf8f22c4af0f1834ee9df81c7dcdfaf238704a984"}
Jul 10 18:43:04.031143 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:04.030789    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66a2f044-d6d4-401c-b49e-19f73a1bb034","Type":"ContainerStarted","Data":"176b8e8f67219bf0cba8cd5f103c40810c759d625e402317de2ce0f9d50b787c"}
Jul 10 18:43:04.031143 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:04.030830    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66a2f044-d6d4-401c-b49e-19f73a1bb034","Type":"ContainerStarted","Data":"5df1679a272a7aa928f8137e66110bef7a65d0a82346a78171f91825f9607294"}
Jul 10 18:43:04.031143 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:04.030898    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66a2f044-d6d4-401c-b49e-19f73a1bb034","Type":"ContainerStarted","Data":"d78000428be9354293360a5ade49d1cfd9764135326916318ff85a36b901ea07"}
Jul 10 18:43:04.031143 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:04.030939    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66a2f044-d6d4-401c-b49e-19f73a1bb034","Type":"ContainerStarted","Data":"7814e2c6be6db4f9916b949fc41402410ac9c5ddc18aa681065853e426f35617"}
Jul 10 18:43:04.068704 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:04.068619    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.053444035 podStartE2EDuration="2.068597327s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.153292ms" isStatefulPod=false podCreationTimestamp="2026-07-10 18:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:43:04.06746197 +0000 UTC m=+207.541554877" watchObservedRunningTime="2026-07-10 18:43:04.068597327 +0000 UTC m=+207.542690213"
Jul 10 18:43:04.945979 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:04.945941    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:43:04.946332 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:04.946293    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="prometheus" containerID="cri-o://bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3" gracePeriod=600
Jul 10 18:43:04.946332 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:04.946314    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="kube-rbac-proxy" containerID="cri-o://b6e6839f22b6e7a6eebe1a0eadb8a3e249728f34d7bec102ce947aae167fdacc" gracePeriod=600
Jul 10 18:43:04.946557 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:04.946340    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="kube-rbac-proxy-web" containerID="cri-o://68b1314deb3ca3e9750582ec3f030d11294f0e75f52d09bce05940a360bd119b" gracePeriod=600
Jul 10 18:43:04.946557 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:04.946358    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://8c2d8e3dce1475802755f3589aa8c166f866de620b30cf298fb11ce43ce1b740" gracePeriod=600
Jul 10 18:43:04.946557 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:04.946374    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="config-reloader" containerID="cri-o://347f89fee1940927f3822eb130dc6ec98c9ed1bfb3c891ae736c6129dfe8b99b" gracePeriod=600
Jul 10 18:43:04.946557 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:04.946343    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="thanos-sidecar" containerID="cri-o://f197c15bc6881922f6b36fd50d1b024e82be25a53fc4ecfa80ec88412142ab12" gracePeriod=600
Jul 10 18:43:06.045857 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:06.045822    2561 generic.go:356] "Generic (PLEG): container finished" podID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerID="68b1314deb3ca3e9750582ec3f030d11294f0e75f52d09bce05940a360bd119b" exitCode=0
Jul 10 18:43:06.045857 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:06.045850    2561 generic.go:356] "Generic (PLEG): container finished" podID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerID="f197c15bc6881922f6b36fd50d1b024e82be25a53fc4ecfa80ec88412142ab12" exitCode=0
Jul 10 18:43:06.045857 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:06.045856    2561 generic.go:356] "Generic (PLEG): container finished" podID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerID="347f89fee1940927f3822eb130dc6ec98c9ed1bfb3c891ae736c6129dfe8b99b" exitCode=0
Jul 10 18:43:06.046334 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:06.045895    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8df723-b571-4680-b235-ab8b02aa0b7a","Type":"ContainerDied","Data":"68b1314deb3ca3e9750582ec3f030d11294f0e75f52d09bce05940a360bd119b"}
Jul 10 18:43:06.046334 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:06.046023    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8df723-b571-4680-b235-ab8b02aa0b7a","Type":"ContainerDied","Data":"f197c15bc6881922f6b36fd50d1b024e82be25a53fc4ecfa80ec88412142ab12"}
Jul 10 18:43:06.046334 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:06.046069    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8df723-b571-4680-b235-ab8b02aa0b7a","Type":"ContainerDied","Data":"347f89fee1940927f3822eb130dc6ec98c9ed1bfb3c891ae736c6129dfe8b99b"}
Jul 10 18:43:06.952032 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:06.951988    2561 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Jul 10 18:43:06.952985 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:06.952949    2561 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Jul 10 18:43:06.953875 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:06.953846    2561 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Jul 10 18:43:06.953949 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:06.953890    2561 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="prometheus" probeResult="unknown"
Jul 10 18:43:07.053193 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:07.053160    2561 generic.go:356] "Generic (PLEG): container finished" podID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerID="8c2d8e3dce1475802755f3589aa8c166f866de620b30cf298fb11ce43ce1b740" exitCode=0
Jul 10 18:43:07.053193 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:07.053187    2561 generic.go:356] "Generic (PLEG): container finished" podID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerID="b6e6839f22b6e7a6eebe1a0eadb8a3e249728f34d7bec102ce947aae167fdacc" exitCode=0
Jul 10 18:43:07.053559 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:07.053240    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8df723-b571-4680-b235-ab8b02aa0b7a","Type":"ContainerDied","Data":"8c2d8e3dce1475802755f3589aa8c166f866de620b30cf298fb11ce43ce1b740"}
Jul 10 18:43:07.053559 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:07.053320    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8df723-b571-4680-b235-ab8b02aa0b7a","Type":"ContainerDied","Data":"b6e6839f22b6e7a6eebe1a0eadb8a3e249728f34d7bec102ce947aae167fdacc"}
Jul 10 18:43:09.808469 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:09.808426    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:09.886634 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:09.886595    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:11.952106 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:11.952034    2561 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Jul 10 18:43:11.953032 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:11.953005    2561 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Jul 10 18:43:11.953859 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:11.953834    2561 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Jul 10 18:43:11.953938 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:11.953873    2561 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="prometheus" probeResult="unknown"
Jul 10 18:43:13.700657 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.700632    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:13.848495 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.848379    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-configmap-metrics-client-ca\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.848495 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.848437    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-trusted-ca-bundle\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.848495 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.848463    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.848495 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.848490    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-prometheus-k8s-tls\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.848822 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.848519    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-kube-rbac-proxy\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.848822 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.848548    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-config\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.848822 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.848660    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-rulefiles-1\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.848822 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.848698    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc8df723-b571-4680-b235-ab8b02aa0b7a-tls-assets\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.848822 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.848742    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-db\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.848822 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.848781    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc8df723-b571-4680-b235-ab8b02aa0b7a-config-out\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.848822 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.848810    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-metrics-client-certs\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.849210 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.848852    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-rulefiles-2\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.849210 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.848883    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-rulefiles-0\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.849210 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.848909    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-configmap-serving-certs-ca-bundle\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.849210 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.848854    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:43:13.849210 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.848950    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-grpc-tls\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.849210 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.848989    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.849210 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.849020    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:43:13.849210 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.849027    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:43:13.849210 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.849029    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvbqg\" (UniqueName: \"kubernetes.io/projected/bc8df723-b571-4680-b235-ab8b02aa0b7a-kube-api-access-gvbqg\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.849210 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.849129    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-configmap-kubelet-serving-ca-bundle\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.849210 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.849168    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-thanos-prometheus-http-client-file\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.849210 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.849194    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-web-config\") pod \"bc8df723-b571-4680-b235-ab8b02aa0b7a\" (UID: \"bc8df723-b571-4680-b235-ab8b02aa0b7a\") "
Jul 10 18:43:13.849791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.849416    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:43:13.849791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.849580    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.849791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.849602    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-configmap-metrics-client-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.849791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.849619    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.849791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.849635    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.850038 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.849969    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:43:13.851356 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.851211    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:43:13.851514 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.851448    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-config" (OuterVolumeSpecName: "config") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:43:13.851577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.851505    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:43:13.851896 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.851865    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:43:13.852097 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.852013    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:43:13.852473 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.852441    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:43:13.852571 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.852451    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:43:13.853410 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.853366    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:43:13.853610 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.853582    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8df723-b571-4680-b235-ab8b02aa0b7a-kube-api-access-gvbqg" (OuterVolumeSpecName: "kube-api-access-gvbqg") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "kube-api-access-gvbqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:43:13.853789 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.853746    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:43:13.853891 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.853841    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:43:13.854330 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.854296    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:43:13.854453 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.854431    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8df723-b571-4680-b235-ab8b02aa0b7a-config-out" (OuterVolumeSpecName: "config-out") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:43:13.854884 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.854860    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8df723-b571-4680-b235-ab8b02aa0b7a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:43:13.864132 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.864071    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-web-config" (OuterVolumeSpecName: "web-config") pod "bc8df723-b571-4680-b235-ab8b02aa0b7a" (UID: "bc8df723-b571-4680-b235-ab8b02aa0b7a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:43:13.950957 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.950908    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvbqg\" (UniqueName: \"kubernetes.io/projected/bc8df723-b571-4680-b235-ab8b02aa0b7a-kube-api-access-gvbqg\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.950957 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.950951    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.950957 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.950962    2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.950957 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.950973    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-web-config\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.951266 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.950984    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.951266 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.950993    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.951266 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.951002    2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-kube-rbac-proxy\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.951266 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.951010    2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-config\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.951266 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.951018    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc8df723-b571-4680-b235-ab8b02aa0b7a-tls-assets\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.951266 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.951025    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-db\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.951266 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.951032    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc8df723-b571-4680-b235-ab8b02aa0b7a-config-out\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.951266 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.951041    2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-metrics-client-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.951266 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.951049    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.951266 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.951059    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc8df723-b571-4680-b235-ab8b02aa0b7a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.951266 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.951069    2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-grpc-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:13.951266 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:13.951101    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc8df723-b571-4680-b235-ab8b02aa0b7a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:43:14.094693 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.094659    2561 generic.go:356] "Generic (PLEG): container finished" podID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerID="bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3" exitCode=0
Jul 10 18:43:14.094842 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.094724    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8df723-b571-4680-b235-ab8b02aa0b7a","Type":"ContainerDied","Data":"bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3"}
Jul 10 18:43:14.094842 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.094742    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.094842 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.094787    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8df723-b571-4680-b235-ab8b02aa0b7a","Type":"ContainerDied","Data":"0d02dda4fac843479a5bd3fd1abb06c1d938f331bbeec225398bfaa134673071"}
Jul 10 18:43:14.094842 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.094823    2561 scope.go:117] "RemoveContainer" containerID="8c2d8e3dce1475802755f3589aa8c166f866de620b30cf298fb11ce43ce1b740"
Jul 10 18:43:14.103159 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.103133    2561 scope.go:117] "RemoveContainer" containerID="b6e6839f22b6e7a6eebe1a0eadb8a3e249728f34d7bec102ce947aae167fdacc"
Jul 10 18:43:14.110214 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.110194    2561 scope.go:117] "RemoveContainer" containerID="68b1314deb3ca3e9750582ec3f030d11294f0e75f52d09bce05940a360bd119b"
Jul 10 18:43:14.116839 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.116819    2561 scope.go:117] "RemoveContainer" containerID="f197c15bc6881922f6b36fd50d1b024e82be25a53fc4ecfa80ec88412142ab12"
Jul 10 18:43:14.123627 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.123516    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:43:14.124257 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.124243    2561 scope.go:117] "RemoveContainer" containerID="347f89fee1940927f3822eb130dc6ec98c9ed1bfb3c891ae736c6129dfe8b99b"
Jul 10 18:43:14.128416 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.128397    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:43:14.133428 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.133411    2561 scope.go:117] "RemoveContainer" containerID="bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3"
Jul 10 18:43:14.140745 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.140726    2561 scope.go:117] "RemoveContainer" containerID="2048337224e2835a367ebab139c276edc99fe70c71506b72b3637beac3516a10"
Jul 10 18:43:14.147461 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.147444    2561 scope.go:117] "RemoveContainer" containerID="8c2d8e3dce1475802755f3589aa8c166f866de620b30cf298fb11ce43ce1b740"
Jul 10 18:43:14.147721 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:14.147699    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2d8e3dce1475802755f3589aa8c166f866de620b30cf298fb11ce43ce1b740\": container with ID starting with 8c2d8e3dce1475802755f3589aa8c166f866de620b30cf298fb11ce43ce1b740 not found: ID does not exist" containerID="8c2d8e3dce1475802755f3589aa8c166f866de620b30cf298fb11ce43ce1b740"
Jul 10 18:43:14.147776 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.147734    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2d8e3dce1475802755f3589aa8c166f866de620b30cf298fb11ce43ce1b740"} err="failed to get container status \"8c2d8e3dce1475802755f3589aa8c166f866de620b30cf298fb11ce43ce1b740\": rpc error: code = NotFound desc = could not find container \"8c2d8e3dce1475802755f3589aa8c166f866de620b30cf298fb11ce43ce1b740\": container with ID starting with 8c2d8e3dce1475802755f3589aa8c166f866de620b30cf298fb11ce43ce1b740 not found: ID does not exist"
Jul 10 18:43:14.147776 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.147754    2561 scope.go:117] "RemoveContainer" containerID="b6e6839f22b6e7a6eebe1a0eadb8a3e249728f34d7bec102ce947aae167fdacc"
Jul 10 18:43:14.148008 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:14.147991    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e6839f22b6e7a6eebe1a0eadb8a3e249728f34d7bec102ce947aae167fdacc\": container with ID starting with b6e6839f22b6e7a6eebe1a0eadb8a3e249728f34d7bec102ce947aae167fdacc not found: ID does not exist" containerID="b6e6839f22b6e7a6eebe1a0eadb8a3e249728f34d7bec102ce947aae167fdacc"
Jul 10 18:43:14.148097 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.148022    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e6839f22b6e7a6eebe1a0eadb8a3e249728f34d7bec102ce947aae167fdacc"} err="failed to get container status \"b6e6839f22b6e7a6eebe1a0eadb8a3e249728f34d7bec102ce947aae167fdacc\": rpc error: code = NotFound desc = could not find container \"b6e6839f22b6e7a6eebe1a0eadb8a3e249728f34d7bec102ce947aae167fdacc\": container with ID starting with b6e6839f22b6e7a6eebe1a0eadb8a3e249728f34d7bec102ce947aae167fdacc not found: ID does not exist"
Jul 10 18:43:14.148097 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.148047    2561 scope.go:117] "RemoveContainer" containerID="68b1314deb3ca3e9750582ec3f030d11294f0e75f52d09bce05940a360bd119b"
Jul 10 18:43:14.148326 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:14.148309    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b1314deb3ca3e9750582ec3f030d11294f0e75f52d09bce05940a360bd119b\": container with ID starting with 68b1314deb3ca3e9750582ec3f030d11294f0e75f52d09bce05940a360bd119b not found: ID does not exist" containerID="68b1314deb3ca3e9750582ec3f030d11294f0e75f52d09bce05940a360bd119b"
Jul 10 18:43:14.148364 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.148332    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b1314deb3ca3e9750582ec3f030d11294f0e75f52d09bce05940a360bd119b"} err="failed to get container status \"68b1314deb3ca3e9750582ec3f030d11294f0e75f52d09bce05940a360bd119b\": rpc error: code = NotFound desc = could not find container \"68b1314deb3ca3e9750582ec3f030d11294f0e75f52d09bce05940a360bd119b\": container with ID starting with 68b1314deb3ca3e9750582ec3f030d11294f0e75f52d09bce05940a360bd119b not found: ID does not exist"
Jul 10 18:43:14.148364 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.148346    2561 scope.go:117] "RemoveContainer" containerID="f197c15bc6881922f6b36fd50d1b024e82be25a53fc4ecfa80ec88412142ab12"
Jul 10 18:43:14.148541 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:14.148526    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f197c15bc6881922f6b36fd50d1b024e82be25a53fc4ecfa80ec88412142ab12\": container with ID starting with f197c15bc6881922f6b36fd50d1b024e82be25a53fc4ecfa80ec88412142ab12 not found: ID does not exist" containerID="f197c15bc6881922f6b36fd50d1b024e82be25a53fc4ecfa80ec88412142ab12"
Jul 10 18:43:14.148583 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.148544    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f197c15bc6881922f6b36fd50d1b024e82be25a53fc4ecfa80ec88412142ab12"} err="failed to get container status \"f197c15bc6881922f6b36fd50d1b024e82be25a53fc4ecfa80ec88412142ab12\": rpc error: code = NotFound desc = could not find container \"f197c15bc6881922f6b36fd50d1b024e82be25a53fc4ecfa80ec88412142ab12\": container with ID starting with f197c15bc6881922f6b36fd50d1b024e82be25a53fc4ecfa80ec88412142ab12 not found: ID does not exist"
Jul 10 18:43:14.148583 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.148557    2561 scope.go:117] "RemoveContainer" containerID="347f89fee1940927f3822eb130dc6ec98c9ed1bfb3c891ae736c6129dfe8b99b"
Jul 10 18:43:14.148740 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:14.148722    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"347f89fee1940927f3822eb130dc6ec98c9ed1bfb3c891ae736c6129dfe8b99b\": container with ID starting with 347f89fee1940927f3822eb130dc6ec98c9ed1bfb3c891ae736c6129dfe8b99b not found: ID does not exist" containerID="347f89fee1940927f3822eb130dc6ec98c9ed1bfb3c891ae736c6129dfe8b99b"
Jul 10 18:43:14.148776 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.148743    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"347f89fee1940927f3822eb130dc6ec98c9ed1bfb3c891ae736c6129dfe8b99b"} err="failed to get container status \"347f89fee1940927f3822eb130dc6ec98c9ed1bfb3c891ae736c6129dfe8b99b\": rpc error: code = NotFound desc = could not find container \"347f89fee1940927f3822eb130dc6ec98c9ed1bfb3c891ae736c6129dfe8b99b\": container with ID starting with 347f89fee1940927f3822eb130dc6ec98c9ed1bfb3c891ae736c6129dfe8b99b not found: ID does not exist"
Jul 10 18:43:14.148776 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.148754    2561 scope.go:117] "RemoveContainer" containerID="bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3"
Jul 10 18:43:14.148970 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:14.148954    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3\": container with ID starting with bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3 not found: ID does not exist" containerID="bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3"
Jul 10 18:43:14.149033 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.148987    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3"} err="failed to get container status \"bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3\": rpc error: code = NotFound desc = could not find container \"bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3\": container with ID starting with bbddabae2b3366846eb8b6ca325053cfcd5959c1619d340422ae1587714579b3 not found: ID does not exist"
Jul 10 18:43:14.149033 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.149009    2561 scope.go:117] "RemoveContainer" containerID="2048337224e2835a367ebab139c276edc99fe70c71506b72b3637beac3516a10"
Jul 10 18:43:14.149265 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:43:14.149249    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2048337224e2835a367ebab139c276edc99fe70c71506b72b3637beac3516a10\": container with ID starting with 2048337224e2835a367ebab139c276edc99fe70c71506b72b3637beac3516a10 not found: ID does not exist" containerID="2048337224e2835a367ebab139c276edc99fe70c71506b72b3637beac3516a10"
Jul 10 18:43:14.149306 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.149269    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2048337224e2835a367ebab139c276edc99fe70c71506b72b3637beac3516a10"} err="failed to get container status \"2048337224e2835a367ebab139c276edc99fe70c71506b72b3637beac3516a10\": rpc error: code = NotFound desc = could not find container \"2048337224e2835a367ebab139c276edc99fe70c71506b72b3637beac3516a10\": container with ID starting with 2048337224e2835a367ebab139c276edc99fe70c71506b72b3637beac3516a10 not found: ID does not exist"
Jul 10 18:43:14.161551 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161521    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:43:14.161829 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161816    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="init-config-reloader"
Jul 10 18:43:14.161920 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161833    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="init-config-reloader"
Jul 10 18:43:14.161920 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161843    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="prometheus"
Jul 10 18:43:14.161920 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161848    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="prometheus"
Jul 10 18:43:14.161920 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161854    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="kube-rbac-proxy-web"
Jul 10 18:43:14.161920 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161861    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="kube-rbac-proxy-web"
Jul 10 18:43:14.161920 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161869    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="config-reloader"
Jul 10 18:43:14.161920 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161874    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="config-reloader"
Jul 10 18:43:14.161920 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161888    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="kube-rbac-proxy"
Jul 10 18:43:14.161920 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161894    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="kube-rbac-proxy"
Jul 10 18:43:14.161920 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161903    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="kube-rbac-proxy-thanos"
Jul 10 18:43:14.161920 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161909    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="kube-rbac-proxy-thanos"
Jul 10 18:43:14.161920 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161915    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="thanos-sidecar"
Jul 10 18:43:14.161920 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161920    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="thanos-sidecar"
Jul 10 18:43:14.162297 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161972    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="thanos-sidecar"
Jul 10 18:43:14.162297 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161980    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="prometheus"
Jul 10 18:43:14.162297 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161987    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="kube-rbac-proxy-web"
Jul 10 18:43:14.162297 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161992    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="kube-rbac-proxy-thanos"
Jul 10 18:43:14.162297 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.161997    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="kube-rbac-proxy"
Jul 10 18:43:14.162297 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.162003    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" containerName="config-reloader"
Jul 10 18:43:14.167175 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.167157    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.172573 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.172547    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 10 18:43:14.174049 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.173677    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 10 18:43:14.174049 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.173679    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 10 18:43:14.174049 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.173750    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 10 18:43:14.174049 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.173758    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 10 18:43:14.174372 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.174352    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 10 18:43:14.174643 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.174626    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 10 18:43:14.174733 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.174627    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 10 18:43:14.174733 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.174626    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 10 18:43:14.174733 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.174716    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-111d5q2uqf0p6\""
Jul 10 18:43:14.174987 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.174967    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 10 18:43:14.175058 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.175008    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 10 18:43:14.175058 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.175051    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2nmhp\""
Jul 10 18:43:14.175187 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.175104    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 10 18:43:14.175343 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.175328    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 10 18:43:14.177155 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.177138    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 10 18:43:14.180746 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.180729    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 10 18:43:14.185042 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.185021    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:43:14.253143 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253105    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253143 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253142    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f89qd\" (UniqueName: \"kubernetes.io/projected/a54a4c89-ddb8-4300-bab5-8fdc94b65977-kube-api-access-f89qd\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253316 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253166    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a54a4c89-ddb8-4300-bab5-8fdc94b65977-config-out\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253316 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253252    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253316 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253287    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-config\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253316 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-web-config\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253450 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253360    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253450 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253391    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a54a4c89-ddb8-4300-bab5-8fdc94b65977-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253450 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253427    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253539 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253539 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253476    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253539 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253497    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253626 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253626 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253595    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253626 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253610    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253714 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253625    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253714 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253640    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253714 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253655    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a54a4c89-ddb8-4300-bab5-8fdc94b65977-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253714 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253698    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.253826 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.253718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.354670 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.354581    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.354670 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.354620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.354670 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.354639    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.354670 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.354662    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.354977 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.354685    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.354977 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.354710    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a54a4c89-ddb8-4300-bab5-8fdc94b65977-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.354977 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.354830    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.354977 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.354875    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.354977 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.354923    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.354977 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.354947    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f89qd\" (UniqueName: \"kubernetes.io/projected/a54a4c89-ddb8-4300-bab5-8fdc94b65977-kube-api-access-f89qd\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.355278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.354980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a54a4c89-ddb8-4300-bab5-8fdc94b65977-config-out\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.355278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.355021    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.355278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.355054    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-config\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.355278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.355099    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-web-config\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.355278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.355127    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.355278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.355155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a54a4c89-ddb8-4300-bab5-8fdc94b65977-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.355278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.355159    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a54a4c89-ddb8-4300-bab5-8fdc94b65977-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.355278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.355179    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.355278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.355220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.355278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.355253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.355278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.355287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.355801 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.355345    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.355801 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.355524    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.357849 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.357815    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.358498 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.358238    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.358498 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.358409    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.358498 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.358440    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a54a4c89-ddb8-4300-bab5-8fdc94b65977-config-out\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.358733 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.358618    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.358733 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.358676    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.358733 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.358702    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a54a4c89-ddb8-4300-bab5-8fdc94b65977-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.359253 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.358989    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-config\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.359253 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.359172    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.359416 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.359331    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.359478 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.359442    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.359707 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.359688    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-web-config\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.359760 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.359728    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.360514 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.360495    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.361155 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.361133    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a54a4c89-ddb8-4300-bab5-8fdc94b65977-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.361311 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.361294    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a54a4c89-ddb8-4300-bab5-8fdc94b65977-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.366359 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.366342    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f89qd\" (UniqueName: \"kubernetes.io/projected/a54a4c89-ddb8-4300-bab5-8fdc94b65977-kube-api-access-f89qd\") pod \"prometheus-k8s-0\" (UID: \"a54a4c89-ddb8-4300-bab5-8fdc94b65977\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.477305 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.477272    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:14.628284 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:14.628257    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:43:14.630339 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:43:14.630315    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54a4c89_ddb8_4300_bab5_8fdc94b65977.slice/crio-d4d0892e08b7e6a788d781d5b40f147fb6ee21ae2a849326cbfa6afeb3d10c8a WatchSource:0}: Error finding container d4d0892e08b7e6a788d781d5b40f147fb6ee21ae2a849326cbfa6afeb3d10c8a: Status 404 returned error can't find the container with id d4d0892e08b7e6a788d781d5b40f147fb6ee21ae2a849326cbfa6afeb3d10c8a
Jul 10 18:43:15.071738 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:15.071700    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc8df723-b571-4680-b235-ab8b02aa0b7a" path="/var/lib/kubelet/pods/bc8df723-b571-4680-b235-ab8b02aa0b7a/volumes"
Jul 10 18:43:15.100591 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:15.100560    2561 generic.go:356] "Generic (PLEG): container finished" podID="a54a4c89-ddb8-4300-bab5-8fdc94b65977" containerID="764ef932b78d781e6456b62aaf4182ba564f887f954ec776d1043d370b1e7511" exitCode=0
Jul 10 18:43:15.100777 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:15.100648    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a54a4c89-ddb8-4300-bab5-8fdc94b65977","Type":"ContainerDied","Data":"764ef932b78d781e6456b62aaf4182ba564f887f954ec776d1043d370b1e7511"}
Jul 10 18:43:15.100777 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:15.100734    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a54a4c89-ddb8-4300-bab5-8fdc94b65977","Type":"ContainerStarted","Data":"d4d0892e08b7e6a788d781d5b40f147fb6ee21ae2a849326cbfa6afeb3d10c8a"}
Jul 10 18:43:16.109482 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:16.109446    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a54a4c89-ddb8-4300-bab5-8fdc94b65977","Type":"ContainerStarted","Data":"ab86607a9957be4e9c207c026b9ef80e542bb6f666ea9e77ed56797767ae7ebc"}
Jul 10 18:43:16.109875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:16.109514    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a54a4c89-ddb8-4300-bab5-8fdc94b65977","Type":"ContainerStarted","Data":"48a9448478de054d0b8d3b5124e59b2783d7c492bc4098252b2dd0196be687da"}
Jul 10 18:43:16.109875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:16.109539    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a54a4c89-ddb8-4300-bab5-8fdc94b65977","Type":"ContainerStarted","Data":"d8877a3bdf5f7f15a88bb7089a8f471731c0b9442776f282ef366e49e7e8d8e6"}
Jul 10 18:43:16.109875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:16.109572    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a54a4c89-ddb8-4300-bab5-8fdc94b65977","Type":"ContainerStarted","Data":"7b71723ac677d4c55ae28763f7f13be7201b587b1bfba7fb206ddcfb991e4510"}
Jul 10 18:43:16.109875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:16.109602    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a54a4c89-ddb8-4300-bab5-8fdc94b65977","Type":"ContainerStarted","Data":"1261aa4ab36200ebc4ac2040b8ad1381dafb31a26fd164cbd51e4d98e27f943d"}
Jul 10 18:43:16.109875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:16.109628    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a54a4c89-ddb8-4300-bab5-8fdc94b65977","Type":"ContainerStarted","Data":"4a1cbf780ccb9e3d41c9d9bab996313d023225602b29e4cf29a44ba76f7a27e9"}
Jul 10 18:43:16.144753 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:16.144702    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.131152159 podStartE2EDuration="2.144679067s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.526908ms" isStatefulPod=false podCreationTimestamp="2026-07-10 18:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:43:16.143687002 +0000 UTC m=+219.617779909" watchObservedRunningTime="2026-07-10 18:43:16.144679067 +0000 UTC m=+219.618771954"
Jul 10 18:43:19.478048 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:19.478009    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:20.322684 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:20.322639    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:20.423282 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:20.423242    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:20.484330 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:20.484297    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:40.907157 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:40.907120    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:41.012765 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:41.012710    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:41.663698 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:43:41.663665    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:44:14.478523 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:14.478475    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:44:14.494013 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:14.493985    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:44:15.360049 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:15.360013    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:44:24.847507 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:24.847473    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:44:24.909216 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:24.909175    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:44:36.780014 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:36.779979    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:44:36.893754 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:36.893719    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:44:36.962095 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:36.962056    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 18:44:36.966949 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:36.966918    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 18:44:36.970577 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:36.970557    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 18:44:36.974784 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:36.974762    2561 kubelet.go:1648] "Image garbage collection succeeded"
Jul 10 18:44:36.978673 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:36.978656    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 18:44:37.819117 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:37.819056    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:44:39.394443 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:39.394402    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:44:39.495704 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:39.495667    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:44:39.559796 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:39.559752    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:44:56.078727 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:56.078686    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:44:59.868988 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:59.868948    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:44:59.923834 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:44:59.923800    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:45:10.673852 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:45:10.673812    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:45:10.737161 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:45:10.737125    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:45:10.792241 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:45:10.792204    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:45:37.671861 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:45:37.671771    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:45:39.482748 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:45:39.482718    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:45:59.771632 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:45:59.771591    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:45:59.870860 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:45:59.870827    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:05.551839 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:46:05.551799    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:05.847330 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:46:05.847233    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:05.986984 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:46:05.986942    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:06.088779 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:46:06.088743    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:06.244540 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:46:06.244503    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:06.421806 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:46:06.421777    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:06.517917 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:46:06.517821    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:06.622628 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:46:06.622594    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:06.824413 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:46:06.824325    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:06.969517 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:46:06.969481    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:07.070210 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:46:07.070178    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:07.170476 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:46:07.170428    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:07.270355 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:46:07.270322    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:59.159474 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:46:59.159327    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:59.258715 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:46:59.258678    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:10.386701 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:10.386658    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:11.156626 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:11.156592    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:14.005879 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:14.005846    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:14.105497 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:14.105463    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:16.049221 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:16.049188    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:16.150709 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:16.150679    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:17.417528 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:17.417487    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:18.419180 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:18.419135    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:18.608737 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:18.608697    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:18.993429 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:18.993392    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:19.046521 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:19.046484    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:19.496336 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:19.496303    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:19.588423 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:19.588391    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:20.611741 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:20.611709    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:20.715729 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:20.715638    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:20.812460 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:20.812425    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:24.878089 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:24.878037    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:24.978665 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:24.978625    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:26.214825 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:26.214789    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:26.408379 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:26.408347    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-w7mhw"]
Jul 10 18:47:26.411622 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:26.411605    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-w7mhw"
Jul 10 18:47:26.414375 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:26.414352    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 10 18:47:26.414956 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:26.414937    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 10 18:47:26.415408 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:26.415392    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-bx9l4\""
Jul 10 18:47:26.425784 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:26.425749    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-w7mhw"]
Jul 10 18:47:26.524356 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:26.524266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv4zj\" (UniqueName: \"kubernetes.io/projected/bd234bc9-2d70-4190-bfd8-6b42641ad3ce-kube-api-access-zv4zj\") pod \"cert-manager-55b4d954d5-w7mhw\" (UID: \"bd234bc9-2d70-4190-bfd8-6b42641ad3ce\") " pod="cert-manager/cert-manager-55b4d954d5-w7mhw"
Jul 10 18:47:26.524511 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:26.524394    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd234bc9-2d70-4190-bfd8-6b42641ad3ce-bound-sa-token\") pod \"cert-manager-55b4d954d5-w7mhw\" (UID: \"bd234bc9-2d70-4190-bfd8-6b42641ad3ce\") " pod="cert-manager/cert-manager-55b4d954d5-w7mhw"
Jul 10 18:47:26.625357 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:26.625317    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv4zj\" (UniqueName: \"kubernetes.io/projected/bd234bc9-2d70-4190-bfd8-6b42641ad3ce-kube-api-access-zv4zj\") pod \"cert-manager-55b4d954d5-w7mhw\" (UID: \"bd234bc9-2d70-4190-bfd8-6b42641ad3ce\") " pod="cert-manager/cert-manager-55b4d954d5-w7mhw"
Jul 10 18:47:26.625546 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:26.625409    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd234bc9-2d70-4190-bfd8-6b42641ad3ce-bound-sa-token\") pod \"cert-manager-55b4d954d5-w7mhw\" (UID: \"bd234bc9-2d70-4190-bfd8-6b42641ad3ce\") " pod="cert-manager/cert-manager-55b4d954d5-w7mhw"
Jul 10 18:47:26.633645 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:26.633618    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd234bc9-2d70-4190-bfd8-6b42641ad3ce-bound-sa-token\") pod \"cert-manager-55b4d954d5-w7mhw\" (UID: \"bd234bc9-2d70-4190-bfd8-6b42641ad3ce\") " pod="cert-manager/cert-manager-55b4d954d5-w7mhw"
Jul 10 18:47:26.633764 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:26.633706    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv4zj\" (UniqueName: \"kubernetes.io/projected/bd234bc9-2d70-4190-bfd8-6b42641ad3ce-kube-api-access-zv4zj\") pod \"cert-manager-55b4d954d5-w7mhw\" (UID: \"bd234bc9-2d70-4190-bfd8-6b42641ad3ce\") " pod="cert-manager/cert-manager-55b4d954d5-w7mhw"
Jul 10 18:47:26.737897 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:26.737857    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-w7mhw"
Jul 10 18:47:26.869367 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:26.869334    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-w7mhw"]
Jul 10 18:47:26.872374 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:47:26.872349    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd234bc9_2d70_4190_bfd8_6b42641ad3ce.slice/crio-cad7ea6291292a58e0c80152fde7752fdf783d9ae1d7baa9e0097e8d78c87f9f WatchSource:0}: Error finding container cad7ea6291292a58e0c80152fde7752fdf783d9ae1d7baa9e0097e8d78c87f9f: Status 404 returned error can't find the container with id cad7ea6291292a58e0c80152fde7752fdf783d9ae1d7baa9e0097e8d78c87f9f
Jul 10 18:47:26.874217 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:26.874200    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 18:47:27.100337 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:27.100247    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-w7mhw" event={"ID":"bd234bc9-2d70-4190-bfd8-6b42641ad3ce","Type":"ContainerStarted","Data":"cad7ea6291292a58e0c80152fde7752fdf783d9ae1d7baa9e0097e8d78c87f9f"}
Jul 10 18:47:30.116675 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:30.116637    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-w7mhw" event={"ID":"bd234bc9-2d70-4190-bfd8-6b42641ad3ce","Type":"ContainerStarted","Data":"a4d1a223eeb6f28e0f458d7a98ade5669602c7ecbda979e7354a04d18372d390"}
Jul 10 18:47:30.150234 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:30.150182    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-w7mhw" podStartSLOduration=1.390511698 podStartE2EDuration="4.150162914s" totalImagesPullingTime="2.759651216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:47:30.148467999 +0000 UTC m=+473.622561095" watchObservedRunningTime="2026-07-10 18:47:30.150162914 +0000 UTC m=+473.624255801"
Jul 10 18:47:38.606924 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:38.606890    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:38.706916 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:47:38.706877    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:04.553187 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:04.553149    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:08.778474 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:08.778439    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:08.848387 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:08.848346    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"]
Jul 10 18:48:08.851685 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:08.851664    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"
Jul 10 18:48:08.855893 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:08.855875    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 10 18:48:08.856154 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:08.856139    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 10 18:48:08.856437 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:08.856423    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 10 18:48:08.856482 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:08.856440    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:48:08.860220 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:08.860201    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-r2nwg\""
Jul 10 18:48:08.860317 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:08.860267    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 10 18:48:08.862353 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:08.862331    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:08.880458 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:08.880427    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"]
Jul 10 18:48:08.984070 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:08.984037    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ccda2068-656d-48a0-9f3d-293cf7124795-metrics-cert\") pod \"lws-controller-manager-849cc6f9d7-48b6z\" (UID: \"ccda2068-656d-48a0-9f3d-293cf7124795\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"
Jul 10 18:48:08.984284 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:08.984173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9j4s\" (UniqueName: \"kubernetes.io/projected/ccda2068-656d-48a0-9f3d-293cf7124795-kube-api-access-m9j4s\") pod \"lws-controller-manager-849cc6f9d7-48b6z\" (UID: \"ccda2068-656d-48a0-9f3d-293cf7124795\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"
Jul 10 18:48:08.984284 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:08.984224    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ccda2068-656d-48a0-9f3d-293cf7124795-manager-config\") pod \"lws-controller-manager-849cc6f9d7-48b6z\" (UID: \"ccda2068-656d-48a0-9f3d-293cf7124795\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"
Jul 10 18:48:08.984284 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:08.984257    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccda2068-656d-48a0-9f3d-293cf7124795-cert\") pod \"lws-controller-manager-849cc6f9d7-48b6z\" (UID: \"ccda2068-656d-48a0-9f3d-293cf7124795\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"
Jul 10 18:48:09.085236 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:09.085143    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9j4s\" (UniqueName: \"kubernetes.io/projected/ccda2068-656d-48a0-9f3d-293cf7124795-kube-api-access-m9j4s\") pod \"lws-controller-manager-849cc6f9d7-48b6z\" (UID: \"ccda2068-656d-48a0-9f3d-293cf7124795\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"
Jul 10 18:48:09.085236 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:09.085187    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ccda2068-656d-48a0-9f3d-293cf7124795-manager-config\") pod \"lws-controller-manager-849cc6f9d7-48b6z\" (UID: \"ccda2068-656d-48a0-9f3d-293cf7124795\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"
Jul 10 18:48:09.085236 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:09.085224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccda2068-656d-48a0-9f3d-293cf7124795-cert\") pod \"lws-controller-manager-849cc6f9d7-48b6z\" (UID: \"ccda2068-656d-48a0-9f3d-293cf7124795\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"
Jul 10 18:48:09.085515 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:09.085278    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ccda2068-656d-48a0-9f3d-293cf7124795-metrics-cert\") pod \"lws-controller-manager-849cc6f9d7-48b6z\" (UID: \"ccda2068-656d-48a0-9f3d-293cf7124795\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"
Jul 10 18:48:09.085770 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:09.085746    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ccda2068-656d-48a0-9f3d-293cf7124795-manager-config\") pod \"lws-controller-manager-849cc6f9d7-48b6z\" (UID: \"ccda2068-656d-48a0-9f3d-293cf7124795\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"
Jul 10 18:48:09.087636 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:09.087612    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccda2068-656d-48a0-9f3d-293cf7124795-cert\") pod \"lws-controller-manager-849cc6f9d7-48b6z\" (UID: \"ccda2068-656d-48a0-9f3d-293cf7124795\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"
Jul 10 18:48:09.087732 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:09.087661    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ccda2068-656d-48a0-9f3d-293cf7124795-metrics-cert\") pod \"lws-controller-manager-849cc6f9d7-48b6z\" (UID: \"ccda2068-656d-48a0-9f3d-293cf7124795\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"
Jul 10 18:48:09.093796 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:09.093766    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9j4s\" (UniqueName: \"kubernetes.io/projected/ccda2068-656d-48a0-9f3d-293cf7124795-kube-api-access-m9j4s\") pod \"lws-controller-manager-849cc6f9d7-48b6z\" (UID: \"ccda2068-656d-48a0-9f3d-293cf7124795\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"
Jul 10 18:48:09.161337 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:09.161295    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"
Jul 10 18:48:09.316863 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:48:09.316835    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccda2068_656d_48a0_9f3d_293cf7124795.slice/crio-16481a8efc38a3892c499b71a577231cae429651595bd599acba6ce424d90b50 WatchSource:0}: Error finding container 16481a8efc38a3892c499b71a577231cae429651595bd599acba6ce424d90b50: Status 404 returned error can't find the container with id 16481a8efc38a3892c499b71a577231cae429651595bd599acba6ce424d90b50
Jul 10 18:48:09.317220 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:09.317195    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"]
Jul 10 18:48:09.540282 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:09.540247    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:09.610510 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:09.610478    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:10.278228 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:10.278184    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z" event={"ID":"ccda2068-656d-48a0-9f3d-293cf7124795","Type":"ContainerStarted","Data":"16481a8efc38a3892c499b71a577231cae429651595bd599acba6ce424d90b50"}
Jul 10 18:48:10.665926 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:10.665886    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:13.795557 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:13.795518    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:13.897867 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:13.897771    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:14.297242 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:14.297197    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z" event={"ID":"ccda2068-656d-48a0-9f3d-293cf7124795","Type":"ContainerStarted","Data":"c27de615409b84167e5513344829bbc3a20706e0d37f900e31dc0ae6ff802e20"}
Jul 10 18:48:14.297414 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:14.297326    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"
Jul 10 18:48:14.316787 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:14.316718    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z" podStartSLOduration=2.072189308 podStartE2EDuration="6.316697843s" totalImagesPullingTime="4.244508535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:48:14.314464427 +0000 UTC m=+517.788557311" watchObservedRunningTime="2026-07-10 18:48:14.316697843 +0000 UTC m=+517.790790729"
Jul 10 18:48:17.720387 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:17.720348    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:17.813641 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:17.813610    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:17.957571 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:17.957490    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:18.008753 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:18.008718    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:20.881031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:20.881001    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:23.205509 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:23.205472    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:23.306346 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:23.306308    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:25.304258 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:25.304230    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-48b6z"
Jul 10 18:48:28.680135 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:28.680096    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:28.780251 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:28.780210    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:28.881021 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:28.880986    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:34.428659 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:34.428627    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:36.476779 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:36.476739    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-tghfc"]
Jul 10 18:48:36.479830 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:36.479811    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-tghfc"
Jul 10 18:48:36.482410 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:36.482388    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 10 18:48:36.483216 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:36.483197    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 10 18:48:36.483311 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:36.483251    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-ksdtg\""
Jul 10 18:48:36.491808 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:36.491781    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-tghfc"]
Jul 10 18:48:36.526939 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:36.526902    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmgfd\" (UniqueName: \"kubernetes.io/projected/b8760a43-d559-4b40-9a46-161389401834-kube-api-access-gmgfd\") pod \"limitador-operator-controller-manager-6f9f468797-tghfc\" (UID: \"b8760a43-d559-4b40-9a46-161389401834\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-tghfc"
Jul 10 18:48:36.627495 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:36.627455    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmgfd\" (UniqueName: \"kubernetes.io/projected/b8760a43-d559-4b40-9a46-161389401834-kube-api-access-gmgfd\") pod \"limitador-operator-controller-manager-6f9f468797-tghfc\" (UID: \"b8760a43-d559-4b40-9a46-161389401834\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-tghfc"
Jul 10 18:48:36.637637 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:36.637607    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmgfd\" (UniqueName: \"kubernetes.io/projected/b8760a43-d559-4b40-9a46-161389401834-kube-api-access-gmgfd\") pod \"limitador-operator-controller-manager-6f9f468797-tghfc\" (UID: \"b8760a43-d559-4b40-9a46-161389401834\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-tghfc"
Jul 10 18:48:36.790369 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:36.790285    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-tghfc"
Jul 10 18:48:36.825053 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:36.825015    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:36.925707 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:36.925627    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-tghfc"]
Jul 10 18:48:36.928578 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:48:36.928551    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8760a43_d559_4b40_9a46_161389401834.slice/crio-533788a6258833a95dd4b90c63c335c9913e5e93b1850c0b7174be46298570e0 WatchSource:0}: Error finding container 533788a6258833a95dd4b90c63c335c9913e5e93b1850c0b7174be46298570e0: Status 404 returned error can't find the container with id 533788a6258833a95dd4b90c63c335c9913e5e93b1850c0b7174be46298570e0
Jul 10 18:48:37.233819 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:37.233787    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:37.390900 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:37.390867    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-tghfc" event={"ID":"b8760a43-d559-4b40-9a46-161389401834","Type":"ContainerStarted","Data":"533788a6258833a95dd4b90c63c335c9913e5e93b1850c0b7174be46298570e0"}
Jul 10 18:48:39.093017 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:39.092972    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:39.179316 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:39.179279    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:39.826719 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:39.826683    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:40.474837 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:40.474797    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:41.708952 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:41.708913    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:41.828507 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:41.828466    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:42.439576 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:42.439545    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:42.688618 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:42.688582    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:42.844190 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:42.844099    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:42.951509 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:42.951473    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:48.039584 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:48.039547    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:48.139691 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:48.139655    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:51.639797 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:51.639748    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:51.739835 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:51.739805    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:52.079774 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.079732    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9n2w"]
Jul 10 18:48:52.087492 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.087457    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:52.088594 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.088576    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9n2w"
Jul 10 18:48:52.091390 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.091356    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 10 18:48:52.091513 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.091369    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 10 18:48:52.091513 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.091500    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-npkqq\""
Jul 10 18:48:52.092136 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.092116    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9n2w"]
Jul 10 18:48:52.151752 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.151716    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c6ad8822-d4e5-4d82-843d-5f97b6847eb6-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9n2w\" (UID: \"c6ad8822-d4e5-4d82-843d-5f97b6847eb6\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9n2w"
Jul 10 18:48:52.151949 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.151791    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6ad8822-d4e5-4d82-843d-5f97b6847eb6-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9n2w\" (UID: \"c6ad8822-d4e5-4d82-843d-5f97b6847eb6\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9n2w"
Jul 10 18:48:52.152039 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.151946    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp87h\" (UniqueName: \"kubernetes.io/projected/c6ad8822-d4e5-4d82-843d-5f97b6847eb6-kube-api-access-cp87h\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9n2w\" (UID: \"c6ad8822-d4e5-4d82-843d-5f97b6847eb6\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9n2w"
Jul 10 18:48:52.227401 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.227364    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:52.252643 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.252606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c6ad8822-d4e5-4d82-843d-5f97b6847eb6-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9n2w\" (UID: \"c6ad8822-d4e5-4d82-843d-5f97b6847eb6\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9n2w"
Jul 10 18:48:52.252805 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.252666    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6ad8822-d4e5-4d82-843d-5f97b6847eb6-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9n2w\" (UID: \"c6ad8822-d4e5-4d82-843d-5f97b6847eb6\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9n2w"
Jul 10 18:48:52.252805 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.252736    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp87h\" (UniqueName: \"kubernetes.io/projected/c6ad8822-d4e5-4d82-843d-5f97b6847eb6-kube-api-access-cp87h\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9n2w\" (UID: \"c6ad8822-d4e5-4d82-843d-5f97b6847eb6\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9n2w"
Jul 10 18:48:52.253348 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.253328    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c6ad8822-d4e5-4d82-843d-5f97b6847eb6-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9n2w\" (UID: \"c6ad8822-d4e5-4d82-843d-5f97b6847eb6\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9n2w"
Jul 10 18:48:52.255117 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.255071    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6ad8822-d4e5-4d82-843d-5f97b6847eb6-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9n2w\" (UID: \"c6ad8822-d4e5-4d82-843d-5f97b6847eb6\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9n2w"
Jul 10 18:48:52.261675 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.261649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp87h\" (UniqueName: \"kubernetes.io/projected/c6ad8822-d4e5-4d82-843d-5f97b6847eb6-kube-api-access-cp87h\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9n2w\" (UID: \"c6ad8822-d4e5-4d82-843d-5f97b6847eb6\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9n2w"
Jul 10 18:48:52.398623 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.398517    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9n2w"
Jul 10 18:48:52.532958 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:52.532920    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9n2w"]
Jul 10 18:48:52.536355 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:48:52.536326    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6ad8822_d4e5_4d82_843d_5f97b6847eb6.slice/crio-be4c72f96295b4a63cfdeb59b02210378dc4ae3e3533a40af6d0d4185ed56fb4 WatchSource:0}: Error finding container be4c72f96295b4a63cfdeb59b02210378dc4ae3e3533a40af6d0d4185ed56fb4: Status 404 returned error can't find the container with id be4c72f96295b4a63cfdeb59b02210378dc4ae3e3533a40af6d0d4185ed56fb4
Jul 10 18:48:53.460601 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:53.460560    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9n2w" event={"ID":"c6ad8822-d4e5-4d82-843d-5f97b6847eb6","Type":"ContainerStarted","Data":"be4c72f96295b4a63cfdeb59b02210378dc4ae3e3533a40af6d0d4185ed56fb4"}
Jul 10 18:48:54.144430 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:54.144391    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:54.246163 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:54.246113    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:58.489320 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:58.489275    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-tghfc" event={"ID":"b8760a43-d559-4b40-9a46-161389401834","Type":"ContainerStarted","Data":"12751470da9e68174702fdeef0579e48b58b18ea9de0e99a9a4c646119144b90"}
Jul 10 18:48:58.489803 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:58.489344    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-tghfc"
Jul 10 18:48:58.490854 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:58.490833    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9n2w" event={"ID":"c6ad8822-d4e5-4d82-843d-5f97b6847eb6","Type":"ContainerStarted","Data":"e3aa9e6b1bca6708c396480a58add36a017c6dde799af08837d52931ad43ea4c"}
Jul 10 18:48:58.508864 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:58.508815    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-tghfc" podStartSLOduration=1.7888249429999998 podStartE2EDuration="22.508801403s" totalImagesPullingTime="20.71997646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:48:58.508668822 +0000 UTC m=+561.982761707" watchObservedRunningTime="2026-07-10 18:48:58.508801403 +0000 UTC m=+561.982894288"
Jul 10 18:48:58.540347 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:48:58.540289    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9n2w" podStartSLOduration=1.416336246 podStartE2EDuration="6.540273019s" totalImagesPullingTime="5.123936773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:48:58.539116514 +0000 UTC m=+562.013209390" watchObservedRunningTime="2026-07-10 18:48:58.540273019 +0000 UTC m=+562.014365904"
Jul 10 18:49:00.582519 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:00.582487    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:00.686540 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:00.686499    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:00.783445 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:00.783408    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:07.226811 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:07.226782    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:07.328459 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:07.328417    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:09.498181 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:09.498152    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-tghfc"
Jul 10 18:49:11.020519 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:11.020483    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:11.072418 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:11.072386    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:11.223811 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:11.223770    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:11.282874 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:11.282774    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:14.842591 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:14.842544    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:14.903831 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:14.903796    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:20.202843 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:20.202801    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:20.256483 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:20.256446    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:20.578969 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:20.578932    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:20.680512 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:20.680482    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:20.853124 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:20.853016    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:32.391789 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:32.391751    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:35.876517 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:35.876478    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"]
Jul 10 18:49:35.876943 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:35.876913    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:35.880122 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:35.880099    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"
Jul 10 18:49:35.882405 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:35.882382    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 10 18:49:35.888572 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:35.888544    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"]
Jul 10 18:49:35.938678 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:35.938648    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2c22982-3b88-4093-9011-38e46f647d37-config-file\") pod \"limitador-limitador-ff68ff5cb-zkphb\" (UID: \"d2c22982-3b88-4093-9011-38e46f647d37\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"
Jul 10 18:49:35.938850 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:35.938693    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxz94\" (UniqueName: \"kubernetes.io/projected/d2c22982-3b88-4093-9011-38e46f647d37-kube-api-access-nxz94\") pod \"limitador-limitador-ff68ff5cb-zkphb\" (UID: \"d2c22982-3b88-4093-9011-38e46f647d37\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"
Jul 10 18:49:35.969864 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:35.969831    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"]
Jul 10 18:49:35.972611 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:35.972564    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:36.030578 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:36.030545    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:36.039679 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:36.039651    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2c22982-3b88-4093-9011-38e46f647d37-config-file\") pod \"limitador-limitador-ff68ff5cb-zkphb\" (UID: \"d2c22982-3b88-4093-9011-38e46f647d37\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"
Jul 10 18:49:36.039796 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:36.039699    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxz94\" (UniqueName: \"kubernetes.io/projected/d2c22982-3b88-4093-9011-38e46f647d37-kube-api-access-nxz94\") pod \"limitador-limitador-ff68ff5cb-zkphb\" (UID: \"d2c22982-3b88-4093-9011-38e46f647d37\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"
Jul 10 18:49:36.040294 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:36.040277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2c22982-3b88-4093-9011-38e46f647d37-config-file\") pod \"limitador-limitador-ff68ff5cb-zkphb\" (UID: \"d2c22982-3b88-4093-9011-38e46f647d37\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"
Jul 10 18:49:36.047798 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:36.047774    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxz94\" (UniqueName: \"kubernetes.io/projected/d2c22982-3b88-4093-9011-38e46f647d37-kube-api-access-nxz94\") pod \"limitador-limitador-ff68ff5cb-zkphb\" (UID: \"d2c22982-3b88-4093-9011-38e46f647d37\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"
Jul 10 18:49:36.190833 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:36.190796    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"
Jul 10 18:49:36.335707 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:36.335670    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"]
Jul 10 18:49:36.339170 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:49:36.339142    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c22982_3b88_4093_9011_38e46f647d37.slice/crio-6d79224e0c50d7e9478be987030e56bc5a6a78bf7e46f83035b051e31380110a WatchSource:0}: Error finding container 6d79224e0c50d7e9478be987030e56bc5a6a78bf7e46f83035b051e31380110a: Status 404 returned error can't find the container with id 6d79224e0c50d7e9478be987030e56bc5a6a78bf7e46f83035b051e31380110a
Jul 10 18:49:36.653890 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:36.653800    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-zkphb" event={"ID":"d2c22982-3b88-4093-9011-38e46f647d37","Type":"ContainerStarted","Data":"6d79224e0c50d7e9478be987030e56bc5a6a78bf7e46f83035b051e31380110a"}
Jul 10 18:49:36.677857 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:36.677824    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:36.761381 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:36.761346    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:36.992634 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:36.992604    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 18:49:36.999091 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:36.999047    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 18:49:36.999631 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:36.999617    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 18:49:37.006173 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:37.006156    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 18:49:40.680101 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:40.680032    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-zkphb" event={"ID":"d2c22982-3b88-4093-9011-38e46f647d37","Type":"ContainerStarted","Data":"2726021bd234315343bf0d0811f13ba89bb83dc98760e82f25ccd61661bffa2a"}
Jul 10 18:49:40.680513 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:40.680134    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"
Jul 10 18:49:40.696926 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:40.696871    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-ff68ff5cb-zkphb" podStartSLOduration=1.830605735 podStartE2EDuration="5.696853902s" totalImagesPullingTime="3.866248167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:49:40.695566183 +0000 UTC m=+604.169659068" watchObservedRunningTime="2026-07-10 18:49:40.696853902 +0000 UTC m=+604.170946787"
Jul 10 18:49:51.688370 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:51.688339    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"
Jul 10 18:49:51.870521 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:51.870481    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"]
Jul 10 18:49:51.870739 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:51.870688    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-ff68ff5cb-zkphb" podUID="d2c22982-3b88-4093-9011-38e46f647d37" containerName="limitador" containerID="cri-o://2726021bd234315343bf0d0811f13ba89bb83dc98760e82f25ccd61661bffa2a" gracePeriod=30
Jul 10 18:49:52.424114 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:52.424071    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"
Jul 10 18:49:52.485299 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:52.485192    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2c22982-3b88-4093-9011-38e46f647d37-config-file\") pod \"d2c22982-3b88-4093-9011-38e46f647d37\" (UID: \"d2c22982-3b88-4093-9011-38e46f647d37\") "
Jul 10 18:49:52.485299 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:52.485277    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxz94\" (UniqueName: \"kubernetes.io/projected/d2c22982-3b88-4093-9011-38e46f647d37-kube-api-access-nxz94\") pod \"d2c22982-3b88-4093-9011-38e46f647d37\" (UID: \"d2c22982-3b88-4093-9011-38e46f647d37\") "
Jul 10 18:49:52.485616 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:52.485591    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c22982-3b88-4093-9011-38e46f647d37-config-file" (OuterVolumeSpecName: "config-file") pod "d2c22982-3b88-4093-9011-38e46f647d37" (UID: "d2c22982-3b88-4093-9011-38e46f647d37"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:49:52.487502 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:52.487476    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c22982-3b88-4093-9011-38e46f647d37-kube-api-access-nxz94" (OuterVolumeSpecName: "kube-api-access-nxz94") pod "d2c22982-3b88-4093-9011-38e46f647d37" (UID: "d2c22982-3b88-4093-9011-38e46f647d37"). InnerVolumeSpecName "kube-api-access-nxz94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:49:52.586623 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:52.586569    2561 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2c22982-3b88-4093-9011-38e46f647d37-config-file\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:49:52.586623 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:52.586617    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxz94\" (UniqueName: \"kubernetes.io/projected/d2c22982-3b88-4093-9011-38e46f647d37-kube-api-access-nxz94\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:49:52.737438 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:52.737335    2561 generic.go:356] "Generic (PLEG): container finished" podID="d2c22982-3b88-4093-9011-38e46f647d37" containerID="2726021bd234315343bf0d0811f13ba89bb83dc98760e82f25ccd61661bffa2a" exitCode=0
Jul 10 18:49:52.737438 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:52.737384    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"
Jul 10 18:49:52.737438 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:52.737429    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-zkphb" event={"ID":"d2c22982-3b88-4093-9011-38e46f647d37","Type":"ContainerDied","Data":"2726021bd234315343bf0d0811f13ba89bb83dc98760e82f25ccd61661bffa2a"}
Jul 10 18:49:52.737927 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:52.737484    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-zkphb" event={"ID":"d2c22982-3b88-4093-9011-38e46f647d37","Type":"ContainerDied","Data":"6d79224e0c50d7e9478be987030e56bc5a6a78bf7e46f83035b051e31380110a"}
Jul 10 18:49:52.737927 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:52.737507    2561 scope.go:117] "RemoveContainer" containerID="2726021bd234315343bf0d0811f13ba89bb83dc98760e82f25ccd61661bffa2a"
Jul 10 18:49:52.746158 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:52.746133    2561 scope.go:117] "RemoveContainer" containerID="2726021bd234315343bf0d0811f13ba89bb83dc98760e82f25ccd61661bffa2a"
Jul 10 18:49:52.746468 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:49:52.746449    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2726021bd234315343bf0d0811f13ba89bb83dc98760e82f25ccd61661bffa2a\": container with ID starting with 2726021bd234315343bf0d0811f13ba89bb83dc98760e82f25ccd61661bffa2a not found: ID does not exist" containerID="2726021bd234315343bf0d0811f13ba89bb83dc98760e82f25ccd61661bffa2a"
Jul 10 18:49:52.746520 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:52.746479    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2726021bd234315343bf0d0811f13ba89bb83dc98760e82f25ccd61661bffa2a"} err="failed to get container status \"2726021bd234315343bf0d0811f13ba89bb83dc98760e82f25ccd61661bffa2a\": rpc error: code = NotFound desc = could not find container \"2726021bd234315343bf0d0811f13ba89bb83dc98760e82f25ccd61661bffa2a\": container with ID starting with 2726021bd234315343bf0d0811f13ba89bb83dc98760e82f25ccd61661bffa2a not found: ID does not exist"
Jul 10 18:49:52.758900 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:52.758870    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"]
Jul 10 18:49:52.762706 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:52.762681    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-zkphb"]
Jul 10 18:49:53.072324 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:53.072232    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2c22982-3b88-4093-9011-38e46f647d37" path="/var/lib/kubelet/pods/d2c22982-3b88-4093-9011-38e46f647d37/volumes"
Jul 10 18:49:57.897790 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:57.897754    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:49:58.266147 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:49:58.266106    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:00.330701 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:00.330619    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:00.404463 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:00.404432    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:06.057039 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:06.056995    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:06.445315 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:06.445272    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:06.545678 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:06.545645    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:21.391240 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.391193    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:21.470124 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.470089    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-659578d74d-npbml"]
Jul 10 18:50:21.470511 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.470494    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c22982-3b88-4093-9011-38e46f647d37" containerName="limitador"
Jul 10 18:50:21.470591 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.470515    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c22982-3b88-4093-9011-38e46f647d37" containerName="limitador"
Jul 10 18:50:21.470647 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.470640    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c22982-3b88-4093-9011-38e46f647d37" containerName="limitador"
Jul 10 18:50:21.485741 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.485708    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-npbml"]
Jul 10 18:50:21.485919 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.485822    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-659578d74d-npbml"
Jul 10 18:50:21.488467 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.488435    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 10 18:50:21.488633 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.488484    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 10 18:50:21.489384 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.489361    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-2nncr\""
Jul 10 18:50:21.489496 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.489385    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 10 18:50:21.491430 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.491402    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:21.492977 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.492953    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"]
Jul 10 18:50:21.496665 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.496644    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"
Jul 10 18:50:21.498819 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.498798    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 10 18:50:21.498905 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.498857    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-xfchk\""
Jul 10 18:50:21.505529 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.505500    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"]
Jul 10 18:50:21.541461 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.541431    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4dpj\" (UniqueName: \"kubernetes.io/projected/54ecad8b-4ff3-48c8-8f93-683e4484bea7-kube-api-access-n4dpj\") pod \"kserve-controller-manager-659578d74d-npbml\" (UID: \"54ecad8b-4ff3-48c8-8f93-683e4484bea7\") " pod="kserve/kserve-controller-manager-659578d74d-npbml"
Jul 10 18:50:21.541618 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.541478    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkkrp\" (UniqueName: \"kubernetes.io/projected/dc33c17a-67ba-451e-8191-79bf71bbfebb-kube-api-access-jkkrp\") pod \"llmisvc-controller-manager-68d9dc5d5-jvjjg\" (UID: \"dc33c17a-67ba-451e-8191-79bf71bbfebb\") " pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"
Jul 10 18:50:21.541618 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.541565    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54ecad8b-4ff3-48c8-8f93-683e4484bea7-cert\") pod \"kserve-controller-manager-659578d74d-npbml\" (UID: \"54ecad8b-4ff3-48c8-8f93-683e4484bea7\") " pod="kserve/kserve-controller-manager-659578d74d-npbml"
Jul 10 18:50:21.541690 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.541630    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc33c17a-67ba-451e-8191-79bf71bbfebb-cert\") pod \"llmisvc-controller-manager-68d9dc5d5-jvjjg\" (UID: \"dc33c17a-67ba-451e-8191-79bf71bbfebb\") " pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"
Jul 10 18:50:21.574382 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.574346    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:21.642914 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.642843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc33c17a-67ba-451e-8191-79bf71bbfebb-cert\") pod \"llmisvc-controller-manager-68d9dc5d5-jvjjg\" (UID: \"dc33c17a-67ba-451e-8191-79bf71bbfebb\") " pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"
Jul 10 18:50:21.642914 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.642896    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4dpj\" (UniqueName: \"kubernetes.io/projected/54ecad8b-4ff3-48c8-8f93-683e4484bea7-kube-api-access-n4dpj\") pod \"kserve-controller-manager-659578d74d-npbml\" (UID: \"54ecad8b-4ff3-48c8-8f93-683e4484bea7\") " pod="kserve/kserve-controller-manager-659578d74d-npbml"
Jul 10 18:50:21.643126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.642923    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkkrp\" (UniqueName: \"kubernetes.io/projected/dc33c17a-67ba-451e-8191-79bf71bbfebb-kube-api-access-jkkrp\") pod \"llmisvc-controller-manager-68d9dc5d5-jvjjg\" (UID: \"dc33c17a-67ba-451e-8191-79bf71bbfebb\") " pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"
Jul 10 18:50:21.643126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.642961    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54ecad8b-4ff3-48c8-8f93-683e4484bea7-cert\") pod \"kserve-controller-manager-659578d74d-npbml\" (UID: \"54ecad8b-4ff3-48c8-8f93-683e4484bea7\") " pod="kserve/kserve-controller-manager-659578d74d-npbml"
Jul 10 18:50:21.643126 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:50:21.643012    2561 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 10 18:50:21.643126 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:50:21.643040    2561 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 10 18:50:21.643126 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:50:21.643098    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ecad8b-4ff3-48c8-8f93-683e4484bea7-cert podName:54ecad8b-4ff3-48c8-8f93-683e4484bea7 nodeName:}" failed. No retries permitted until 2026-07-10 18:50:22.143069182 +0000 UTC m=+645.617162045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/54ecad8b-4ff3-48c8-8f93-683e4484bea7-cert") pod "kserve-controller-manager-659578d74d-npbml" (UID: "54ecad8b-4ff3-48c8-8f93-683e4484bea7") : secret "kserve-webhook-server-cert" not found
Jul 10 18:50:21.643126 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:50:21.643110    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc33c17a-67ba-451e-8191-79bf71bbfebb-cert podName:dc33c17a-67ba-451e-8191-79bf71bbfebb nodeName:}" failed. No retries permitted until 2026-07-10 18:50:22.143104626 +0000 UTC m=+645.617197489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/dc33c17a-67ba-451e-8191-79bf71bbfebb-cert") pod "llmisvc-controller-manager-68d9dc5d5-jvjjg" (UID: "dc33c17a-67ba-451e-8191-79bf71bbfebb") : secret "llmisvc-webhook-server-cert" not found
Jul 10 18:50:21.651997 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.651970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkkrp\" (UniqueName: \"kubernetes.io/projected/dc33c17a-67ba-451e-8191-79bf71bbfebb-kube-api-access-jkkrp\") pod \"llmisvc-controller-manager-68d9dc5d5-jvjjg\" (UID: \"dc33c17a-67ba-451e-8191-79bf71bbfebb\") " pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"
Jul 10 18:50:21.652919 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.652902    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4dpj\" (UniqueName: \"kubernetes.io/projected/54ecad8b-4ff3-48c8-8f93-683e4484bea7-kube-api-access-n4dpj\") pod \"kserve-controller-manager-659578d74d-npbml\" (UID: \"54ecad8b-4ff3-48c8-8f93-683e4484bea7\") " pod="kserve/kserve-controller-manager-659578d74d-npbml"
Jul 10 18:50:21.739807 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.739767    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:21.840617 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:21.840580    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:22.147589 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:22.147551    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54ecad8b-4ff3-48c8-8f93-683e4484bea7-cert\") pod \"kserve-controller-manager-659578d74d-npbml\" (UID: \"54ecad8b-4ff3-48c8-8f93-683e4484bea7\") " pod="kserve/kserve-controller-manager-659578d74d-npbml"
Jul 10 18:50:22.147778 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:22.147607    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc33c17a-67ba-451e-8191-79bf71bbfebb-cert\") pod \"llmisvc-controller-manager-68d9dc5d5-jvjjg\" (UID: \"dc33c17a-67ba-451e-8191-79bf71bbfebb\") " pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"
Jul 10 18:50:22.150239 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:22.150212    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54ecad8b-4ff3-48c8-8f93-683e4484bea7-cert\") pod \"kserve-controller-manager-659578d74d-npbml\" (UID: \"54ecad8b-4ff3-48c8-8f93-683e4484bea7\") " pod="kserve/kserve-controller-manager-659578d74d-npbml"
Jul 10 18:50:22.150347 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:22.150212    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc33c17a-67ba-451e-8191-79bf71bbfebb-cert\") pod \"llmisvc-controller-manager-68d9dc5d5-jvjjg\" (UID: \"dc33c17a-67ba-451e-8191-79bf71bbfebb\") " pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"
Jul 10 18:50:22.207194 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:22.207150    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:22.302959 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:22.302917    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:22.398059 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:22.397958    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-659578d74d-npbml"
Jul 10 18:50:22.406372 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:22.406343    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"
Jul 10 18:50:22.551258 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:22.551226    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-npbml"]
Jul 10 18:50:22.553264 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:50:22.553235    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ecad8b_4ff3_48c8_8f93_683e4484bea7.slice/crio-ac34c0fdf7a887fad8cf9d4d9cce583464029e35bd60012a98dd2bbe7f69e0fd WatchSource:0}: Error finding container ac34c0fdf7a887fad8cf9d4d9cce583464029e35bd60012a98dd2bbe7f69e0fd: Status 404 returned error can't find the container with id ac34c0fdf7a887fad8cf9d4d9cce583464029e35bd60012a98dd2bbe7f69e0fd
Jul 10 18:50:22.573152 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:22.573123    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"]
Jul 10 18:50:22.576214 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:50:22.576188    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddc33c17a_67ba_451e_8191_79bf71bbfebb.slice/crio-29b6cfaeec892add525cb775ee03d2d6fc1269271a8e959a73aecbc5541fa375 WatchSource:0}: Error finding container 29b6cfaeec892add525cb775ee03d2d6fc1269271a8e959a73aecbc5541fa375: Status 404 returned error can't find the container with id 29b6cfaeec892add525cb775ee03d2d6fc1269271a8e959a73aecbc5541fa375
Jul 10 18:50:22.872522 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:22.872490    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-659578d74d-npbml" event={"ID":"54ecad8b-4ff3-48c8-8f93-683e4484bea7","Type":"ContainerStarted","Data":"ac34c0fdf7a887fad8cf9d4d9cce583464029e35bd60012a98dd2bbe7f69e0fd"}
Jul 10 18:50:22.873639 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:22.873614    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg" event={"ID":"dc33c17a-67ba-451e-8191-79bf71bbfebb","Type":"ContainerStarted","Data":"29b6cfaeec892add525cb775ee03d2d6fc1269271a8e959a73aecbc5541fa375"}
Jul 10 18:50:23.245329 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:23.245293    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:25.892504 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:25.892476    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-659578d74d-npbml" event={"ID":"54ecad8b-4ff3-48c8-8f93-683e4484bea7","Type":"ContainerStarted","Data":"55728d37d5e301e3f9b79a7fd941fbe89df6532bb4797b54cd9d6d3df4033e87"}
Jul 10 18:50:25.892925 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:25.892609    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-659578d74d-npbml"
Jul 10 18:50:25.914020 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:25.913974    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-659578d74d-npbml" podStartSLOduration=2.284640005 podStartE2EDuration="4.913959341s" totalImagesPullingTime="2.629319336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:50:25.912096451 +0000 UTC m=+649.386189352" watchObservedRunningTime="2026-07-10 18:50:25.913959341 +0000 UTC m=+649.388052226"
Jul 10 18:50:26.899216 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:26.899179    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg" event={"ID":"dc33c17a-67ba-451e-8191-79bf71bbfebb","Type":"ContainerStarted","Data":"46ff2169ebd7999c2e8876784f89efbb6f6834216b5040324416b982d708087b"}
Jul 10 18:50:26.899613 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:26.899452    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"
Jul 10 18:50:26.920545 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:26.920489    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg" podStartSLOduration=2.652299646 podStartE2EDuration="5.920475392s" totalImagesPullingTime="3.268175746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:50:26.919448083 +0000 UTC m=+650.393540970" watchObservedRunningTime="2026-07-10 18:50:26.920475392 +0000 UTC m=+650.394568319"
Jul 10 18:50:32.461639 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:32.461599    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:56.904205 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:56.904172    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-659578d74d-npbml"
Jul 10 18:50:57.906693 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:57.906663    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"
Jul 10 18:50:59.387610 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.387576    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-npbml"]
Jul 10 18:50:59.388067 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.387757    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-659578d74d-npbml" podUID="54ecad8b-4ff3-48c8-8f93-683e4484bea7" containerName="manager" containerID="cri-o://55728d37d5e301e3f9b79a7fd941fbe89df6532bb4797b54cd9d6d3df4033e87" gracePeriod=10
Jul 10 18:50:59.411413 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.411380    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-659578d74d-bkptw"]
Jul 10 18:50:59.415280 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.415261    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-659578d74d-bkptw"
Jul 10 18:50:59.424759 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.424730    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-bkptw"]
Jul 10 18:50:59.583759 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.583727    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f9b899e-e2b5-4c68-a938-feaf6ed5e1e2-cert\") pod \"kserve-controller-manager-659578d74d-bkptw\" (UID: \"2f9b899e-e2b5-4c68-a938-feaf6ed5e1e2\") " pod="kserve/kserve-controller-manager-659578d74d-bkptw"
Jul 10 18:50:59.583927 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.583766    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vpb7\" (UniqueName: \"kubernetes.io/projected/2f9b899e-e2b5-4c68-a938-feaf6ed5e1e2-kube-api-access-8vpb7\") pod \"kserve-controller-manager-659578d74d-bkptw\" (UID: \"2f9b899e-e2b5-4c68-a938-feaf6ed5e1e2\") " pod="kserve/kserve-controller-manager-659578d74d-bkptw"
Jul 10 18:50:59.636551 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.636523    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-659578d74d-npbml"
Jul 10 18:50:59.684508 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.684481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f9b899e-e2b5-4c68-a938-feaf6ed5e1e2-cert\") pod \"kserve-controller-manager-659578d74d-bkptw\" (UID: \"2f9b899e-e2b5-4c68-a938-feaf6ed5e1e2\") " pod="kserve/kserve-controller-manager-659578d74d-bkptw"
Jul 10 18:50:59.684508 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.684513    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vpb7\" (UniqueName: \"kubernetes.io/projected/2f9b899e-e2b5-4c68-a938-feaf6ed5e1e2-kube-api-access-8vpb7\") pod \"kserve-controller-manager-659578d74d-bkptw\" (UID: \"2f9b899e-e2b5-4c68-a938-feaf6ed5e1e2\") " pod="kserve/kserve-controller-manager-659578d74d-bkptw"
Jul 10 18:50:59.687057 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.687022    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f9b899e-e2b5-4c68-a938-feaf6ed5e1e2-cert\") pod \"kserve-controller-manager-659578d74d-bkptw\" (UID: \"2f9b899e-e2b5-4c68-a938-feaf6ed5e1e2\") " pod="kserve/kserve-controller-manager-659578d74d-bkptw"
Jul 10 18:50:59.694068 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.694045    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vpb7\" (UniqueName: \"kubernetes.io/projected/2f9b899e-e2b5-4c68-a938-feaf6ed5e1e2-kube-api-access-8vpb7\") pod \"kserve-controller-manager-659578d74d-bkptw\" (UID: \"2f9b899e-e2b5-4c68-a938-feaf6ed5e1e2\") " pod="kserve/kserve-controller-manager-659578d74d-bkptw"
Jul 10 18:50:59.778950 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.778917    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-659578d74d-bkptw"
Jul 10 18:50:59.785223 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.785195    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4dpj\" (UniqueName: \"kubernetes.io/projected/54ecad8b-4ff3-48c8-8f93-683e4484bea7-kube-api-access-n4dpj\") pod \"54ecad8b-4ff3-48c8-8f93-683e4484bea7\" (UID: \"54ecad8b-4ff3-48c8-8f93-683e4484bea7\") "
Jul 10 18:50:59.785367 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.785253    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54ecad8b-4ff3-48c8-8f93-683e4484bea7-cert\") pod \"54ecad8b-4ff3-48c8-8f93-683e4484bea7\" (UID: \"54ecad8b-4ff3-48c8-8f93-683e4484bea7\") "
Jul 10 18:50:59.787461 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.787435    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ecad8b-4ff3-48c8-8f93-683e4484bea7-kube-api-access-n4dpj" (OuterVolumeSpecName: "kube-api-access-n4dpj") pod "54ecad8b-4ff3-48c8-8f93-683e4484bea7" (UID: "54ecad8b-4ff3-48c8-8f93-683e4484bea7"). InnerVolumeSpecName "kube-api-access-n4dpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:50:59.787461 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.787456    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ecad8b-4ff3-48c8-8f93-683e4484bea7-cert" (OuterVolumeSpecName: "cert") pod "54ecad8b-4ff3-48c8-8f93-683e4484bea7" (UID: "54ecad8b-4ff3-48c8-8f93-683e4484bea7"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:50:59.886867 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.886833    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4dpj\" (UniqueName: \"kubernetes.io/projected/54ecad8b-4ff3-48c8-8f93-683e4484bea7-kube-api-access-n4dpj\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:50:59.886867 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.886864    2561 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54ecad8b-4ff3-48c8-8f93-683e4484bea7-cert\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:50:59.932704 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:50:59.932666    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-bkptw"]
Jul 10 18:50:59.935990 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:50:59.935919    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9b899e_e2b5_4c68_a938_feaf6ed5e1e2.slice/crio-1d3bbac8f81a2b991217421cce5938bce945f62c34e0e9854f5725c8ea5dfaad WatchSource:0}: Error finding container 1d3bbac8f81a2b991217421cce5938bce945f62c34e0e9854f5725c8ea5dfaad: Status 404 returned error can't find the container with id 1d3bbac8f81a2b991217421cce5938bce945f62c34e0e9854f5725c8ea5dfaad
Jul 10 18:51:00.051298 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:00.051262    2561 generic.go:356] "Generic (PLEG): container finished" podID="54ecad8b-4ff3-48c8-8f93-683e4484bea7" containerID="55728d37d5e301e3f9b79a7fd941fbe89df6532bb4797b54cd9d6d3df4033e87" exitCode=0
Jul 10 18:51:00.051503 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:00.051318    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-659578d74d-npbml"
Jul 10 18:51:00.051503 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:00.051359    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-659578d74d-npbml" event={"ID":"54ecad8b-4ff3-48c8-8f93-683e4484bea7","Type":"ContainerDied","Data":"55728d37d5e301e3f9b79a7fd941fbe89df6532bb4797b54cd9d6d3df4033e87"}
Jul 10 18:51:00.051503 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:00.051416    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-659578d74d-npbml" event={"ID":"54ecad8b-4ff3-48c8-8f93-683e4484bea7","Type":"ContainerDied","Data":"ac34c0fdf7a887fad8cf9d4d9cce583464029e35bd60012a98dd2bbe7f69e0fd"}
Jul 10 18:51:00.051503 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:00.051444    2561 scope.go:117] "RemoveContainer" containerID="55728d37d5e301e3f9b79a7fd941fbe89df6532bb4797b54cd9d6d3df4033e87"
Jul 10 18:51:00.052678 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:00.052651    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-659578d74d-bkptw" event={"ID":"2f9b899e-e2b5-4c68-a938-feaf6ed5e1e2","Type":"ContainerStarted","Data":"1d3bbac8f81a2b991217421cce5938bce945f62c34e0e9854f5725c8ea5dfaad"}
Jul 10 18:51:00.060472 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:00.060449    2561 scope.go:117] "RemoveContainer" containerID="55728d37d5e301e3f9b79a7fd941fbe89df6532bb4797b54cd9d6d3df4033e87"
Jul 10 18:51:00.060734 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:51:00.060714    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55728d37d5e301e3f9b79a7fd941fbe89df6532bb4797b54cd9d6d3df4033e87\": container with ID starting with 55728d37d5e301e3f9b79a7fd941fbe89df6532bb4797b54cd9d6d3df4033e87 not found: ID does not exist" containerID="55728d37d5e301e3f9b79a7fd941fbe89df6532bb4797b54cd9d6d3df4033e87"
Jul 10 18:51:00.060778 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:00.060744    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55728d37d5e301e3f9b79a7fd941fbe89df6532bb4797b54cd9d6d3df4033e87"} err="failed to get container status \"55728d37d5e301e3f9b79a7fd941fbe89df6532bb4797b54cd9d6d3df4033e87\": rpc error: code = NotFound desc = could not find container \"55728d37d5e301e3f9b79a7fd941fbe89df6532bb4797b54cd9d6d3df4033e87\": container with ID starting with 55728d37d5e301e3f9b79a7fd941fbe89df6532bb4797b54cd9d6d3df4033e87 not found: ID does not exist"
Jul 10 18:51:00.074007 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:00.073982    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-npbml"]
Jul 10 18:51:00.077760 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:00.077738    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-npbml"]
Jul 10 18:51:01.059016 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:01.058980    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-659578d74d-bkptw" event={"ID":"2f9b899e-e2b5-4c68-a938-feaf6ed5e1e2","Type":"ContainerStarted","Data":"8f88b19aff573e6125210ed863a38ec47b3d71ed2edd014d0aa338d8525efc72"}
Jul 10 18:51:01.059488 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:01.059050    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-659578d74d-bkptw"
Jul 10 18:51:01.071691 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:01.071662    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54ecad8b-4ff3-48c8-8f93-683e4484bea7" path="/var/lib/kubelet/pods/54ecad8b-4ff3-48c8-8f93-683e4484bea7/volumes"
Jul 10 18:51:01.078710 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:01.078635    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-659578d74d-bkptw" podStartSLOduration=1.659619294 podStartE2EDuration="2.07861766s" totalImagesPullingTime="418.998366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:51:01.077576298 +0000 UTC m=+684.551669185" watchObservedRunningTime="2026-07-10 18:51:01.07861766 +0000 UTC m=+684.552710545"
Jul 10 18:51:11.086315 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:11.086279    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:51:32.072169 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:32.072088    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-659578d74d-bkptw"
Jul 10 18:51:33.037637 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.037599    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:51:33.043004 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.042978    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-2488h"]
Jul 10 18:51:33.043408 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.043393    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ecad8b-4ff3-48c8-8f93-683e4484bea7" containerName="manager"
Jul 10 18:51:33.043449 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.043413    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ecad8b-4ff3-48c8-8f93-683e4484bea7" containerName="manager"
Jul 10 18:51:33.043484 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.043478    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ecad8b-4ff3-48c8-8f93-683e4484bea7" containerName="manager"
Jul 10 18:51:33.046426 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.046409    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-2488h"
Jul 10 18:51:33.049722 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.049702    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 10 18:51:33.049846 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.049704    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-7dppv\""
Jul 10 18:51:33.061813 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.061775    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-2488h"]
Jul 10 18:51:33.065790 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.065757    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-ks2cj"]
Jul 10 18:51:33.069401 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.069377    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-ks2cj"
Jul 10 18:51:33.071701 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.071683    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-576k6\""
Jul 10 18:51:33.071871 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.071851    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 10 18:51:33.075991 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.075960    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-ks2cj"]
Jul 10 18:51:33.142395 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.142365    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:51:33.164186 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.164155    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggfhw\" (UniqueName: \"kubernetes.io/projected/03c1aff7-9cff-4589-9773-92b477ff7d9e-kube-api-access-ggfhw\") pod \"odh-model-controller-84b54b7f97-ks2cj\" (UID: \"03c1aff7-9cff-4589-9773-92b477ff7d9e\") " pod="kserve/odh-model-controller-84b54b7f97-ks2cj"
Jul 10 18:51:33.164344 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.164209    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03c1aff7-9cff-4589-9773-92b477ff7d9e-cert\") pod \"odh-model-controller-84b54b7f97-ks2cj\" (UID: \"03c1aff7-9cff-4589-9773-92b477ff7d9e\") " pod="kserve/odh-model-controller-84b54b7f97-ks2cj"
Jul 10 18:51:33.164344 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.164244    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ef5227-5997-4a67-9e3a-c6c88b50d3ed-tls-certs\") pod \"model-serving-api-fd65d7d6-2488h\" (UID: \"e3ef5227-5997-4a67-9e3a-c6c88b50d3ed\") " pod="kserve/model-serving-api-fd65d7d6-2488h"
Jul 10 18:51:33.164344 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.164287    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85q86\" (UniqueName: \"kubernetes.io/projected/e3ef5227-5997-4a67-9e3a-c6c88b50d3ed-kube-api-access-85q86\") pod \"model-serving-api-fd65d7d6-2488h\" (UID: \"e3ef5227-5997-4a67-9e3a-c6c88b50d3ed\") " pod="kserve/model-serving-api-fd65d7d6-2488h"
Jul 10 18:51:33.265566 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.265530    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggfhw\" (UniqueName: \"kubernetes.io/projected/03c1aff7-9cff-4589-9773-92b477ff7d9e-kube-api-access-ggfhw\") pod \"odh-model-controller-84b54b7f97-ks2cj\" (UID: \"03c1aff7-9cff-4589-9773-92b477ff7d9e\") " pod="kserve/odh-model-controller-84b54b7f97-ks2cj"
Jul 10 18:51:33.265566 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.265580    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03c1aff7-9cff-4589-9773-92b477ff7d9e-cert\") pod \"odh-model-controller-84b54b7f97-ks2cj\" (UID: \"03c1aff7-9cff-4589-9773-92b477ff7d9e\") " pod="kserve/odh-model-controller-84b54b7f97-ks2cj"
Jul 10 18:51:33.265843 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.265622    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ef5227-5997-4a67-9e3a-c6c88b50d3ed-tls-certs\") pod \"model-serving-api-fd65d7d6-2488h\" (UID: \"e3ef5227-5997-4a67-9e3a-c6c88b50d3ed\") " pod="kserve/model-serving-api-fd65d7d6-2488h"
Jul 10 18:51:33.265843 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.265649    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85q86\" (UniqueName: \"kubernetes.io/projected/e3ef5227-5997-4a67-9e3a-c6c88b50d3ed-kube-api-access-85q86\") pod \"model-serving-api-fd65d7d6-2488h\" (UID: \"e3ef5227-5997-4a67-9e3a-c6c88b50d3ed\") " pod="kserve/model-serving-api-fd65d7d6-2488h"
Jul 10 18:51:33.268208 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.268179    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03c1aff7-9cff-4589-9773-92b477ff7d9e-cert\") pod \"odh-model-controller-84b54b7f97-ks2cj\" (UID: \"03c1aff7-9cff-4589-9773-92b477ff7d9e\") " pod="kserve/odh-model-controller-84b54b7f97-ks2cj"
Jul 10 18:51:33.268208 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.268198    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ef5227-5997-4a67-9e3a-c6c88b50d3ed-tls-certs\") pod \"model-serving-api-fd65d7d6-2488h\" (UID: \"e3ef5227-5997-4a67-9e3a-c6c88b50d3ed\") " pod="kserve/model-serving-api-fd65d7d6-2488h"
Jul 10 18:51:33.273813 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.273789    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggfhw\" (UniqueName: \"kubernetes.io/projected/03c1aff7-9cff-4589-9773-92b477ff7d9e-kube-api-access-ggfhw\") pod \"odh-model-controller-84b54b7f97-ks2cj\" (UID: \"03c1aff7-9cff-4589-9773-92b477ff7d9e\") " pod="kserve/odh-model-controller-84b54b7f97-ks2cj"
Jul 10 18:51:33.273921 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.273820    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85q86\" (UniqueName: \"kubernetes.io/projected/e3ef5227-5997-4a67-9e3a-c6c88b50d3ed-kube-api-access-85q86\") pod \"model-serving-api-fd65d7d6-2488h\" (UID: \"e3ef5227-5997-4a67-9e3a-c6c88b50d3ed\") " pod="kserve/model-serving-api-fd65d7d6-2488h"
Jul 10 18:51:33.357905 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.357804    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-2488h"
Jul 10 18:51:33.382062 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.382032    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-ks2cj"
Jul 10 18:51:33.510598 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.510416    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-2488h"]
Jul 10 18:51:33.513724 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:51:33.513694    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ef5227_5997_4a67_9e3a_c6c88b50d3ed.slice/crio-ce4db124d92b4e3b02aed24784b61c5c05e0941364227c26d702fd5e45552e7c WatchSource:0}: Error finding container ce4db124d92b4e3b02aed24784b61c5c05e0941364227c26d702fd5e45552e7c: Status 404 returned error can't find the container with id ce4db124d92b4e3b02aed24784b61c5c05e0941364227c26d702fd5e45552e7c
Jul 10 18:51:33.536559 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.536535    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-ks2cj"]
Jul 10 18:51:33.539066 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:51:33.539043    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c1aff7_9cff_4589_9773_92b477ff7d9e.slice/crio-ec1babb5396b373cfdb49fe7bfbb4fb03adcabc7a8afde033dbcdb8966aef4a9 WatchSource:0}: Error finding container ec1babb5396b373cfdb49fe7bfbb4fb03adcabc7a8afde033dbcdb8966aef4a9: Status 404 returned error can't find the container with id ec1babb5396b373cfdb49fe7bfbb4fb03adcabc7a8afde033dbcdb8966aef4a9
Jul 10 18:51:33.892664 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.892619    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:51:33.993772 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:33.993734    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:51:34.222965 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:34.222922    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-ks2cj" event={"ID":"03c1aff7-9cff-4589-9773-92b477ff7d9e","Type":"ContainerStarted","Data":"ec1babb5396b373cfdb49fe7bfbb4fb03adcabc7a8afde033dbcdb8966aef4a9"}
Jul 10 18:51:34.224583 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:34.224552    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-2488h" event={"ID":"e3ef5227-5997-4a67-9e3a-c6c88b50d3ed","Type":"ContainerStarted","Data":"ce4db124d92b4e3b02aed24784b61c5c05e0941364227c26d702fd5e45552e7c"}
Jul 10 18:51:37.244876 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:37.244839    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-2488h" event={"ID":"e3ef5227-5997-4a67-9e3a-c6c88b50d3ed","Type":"ContainerStarted","Data":"cb5d41beb93b689ebb9273c66f4ccfe7dfa4171d2b2aff778ca101d869e5d376"}
Jul 10 18:51:37.245317 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:37.245001    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-2488h"
Jul 10 18:51:37.246278 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:37.246259    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-ks2cj" event={"ID":"03c1aff7-9cff-4589-9773-92b477ff7d9e","Type":"ContainerStarted","Data":"347227a79843030f59196392d7b09131e396af94e01c668e5b797c68410bf767"}
Jul 10 18:51:37.246388 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:37.246374    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-ks2cj"
Jul 10 18:51:37.265997 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:37.265938    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-2488h" podStartSLOduration=1.4067188640000001 podStartE2EDuration="4.265917394s" totalImagesPullingTime="2.85919853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:51:37.265030882 +0000 UTC m=+720.739123803" watchObservedRunningTime="2026-07-10 18:51:37.265917394 +0000 UTC m=+720.740010282"
Jul 10 18:51:37.285128 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:37.284985    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-ks2cj" podStartSLOduration=1.396711351 podStartE2EDuration="4.284969445s" totalImagesPullingTime="2.888258094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:51:37.284280369 +0000 UTC m=+720.758373250" watchObservedRunningTime="2026-07-10 18:51:37.284969445 +0000 UTC m=+720.759062330"
Jul 10 18:51:48.253703 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:48.253627    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-ks2cj"
Jul 10 18:51:48.255707 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:51:48.255687    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-2488h"
Jul 10 18:52:00.394356 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:00.394315    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:52:00.617435 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:00.617397    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:52:10.321589 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:10.321559    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:52:10.470216 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:10.470179    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:52:14.063621 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:14.063586    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:52:14.180894 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:14.180847    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:52:45.696432 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:45.696381    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:52:46.014011 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.013885    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"]
Jul 10 18:52:46.023791 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.023745    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.032625 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.032593    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 18:52:46.032625 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.032606    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-mskkz\""
Jul 10 18:52:46.032860 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.032719    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 18:52:46.033262 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.033239    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jjdpb\""
Jul 10 18:52:46.033354 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.033308    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 10 18:52:46.053666 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.053630    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"]
Jul 10 18:52:46.111940 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.111896    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7xn\" (UniqueName: \"kubernetes.io/projected/3560cbd5-853e-431f-b1d3-472684d59a3d-kube-api-access-qp7xn\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.112126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.112004    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3560cbd5-853e-431f-b1d3-472684d59a3d-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.112126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.112067    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.112126 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.112113    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.112262 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.112153    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.112262 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.112199    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.119818 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.119782    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:52:46.213292 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.213259    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.213464 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.213308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.213464 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.213353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7xn\" (UniqueName: \"kubernetes.io/projected/3560cbd5-853e-431f-b1d3-472684d59a3d-kube-api-access-qp7xn\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.213464 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.213429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3560cbd5-853e-431f-b1d3-472684d59a3d-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.213635 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.213477    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.213635 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.213611    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.213763 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.213672    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.213821 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.213759    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.213911 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.213893    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.213948 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.213934    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.215763 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.215746    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3560cbd5-853e-431f-b1d3-472684d59a3d-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.221429 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.221405    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7xn\" (UniqueName: \"kubernetes.io/projected/3560cbd5-853e-431f-b1d3-472684d59a3d-kube-api-access-qp7xn\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.334834 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.334742    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:52:46.487219 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.487133    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"]
Jul 10 18:52:46.489415 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:52:46.489388    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3560cbd5_853e_431f_b1d3_472684d59a3d.slice/crio-1afe773d67b22d3acbc7537855b6ff4c9014d561c3e31c1425143741ec67d66a WatchSource:0}: Error finding container 1afe773d67b22d3acbc7537855b6ff4c9014d561c3e31c1425143741ec67d66a: Status 404 returned error can't find the container with id 1afe773d67b22d3acbc7537855b6ff4c9014d561c3e31c1425143741ec67d66a
Jul 10 18:52:46.491521 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.491503    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 18:52:46.573712 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:46.573675    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k" event={"ID":"3560cbd5-853e-431f-b1d3-472684d59a3d","Type":"ContainerStarted","Data":"1afe773d67b22d3acbc7537855b6ff4c9014d561c3e31c1425143741ec67d66a"}
Jul 10 18:52:49.590134 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:49.590091    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k" event={"ID":"3560cbd5-853e-431f-b1d3-472684d59a3d","Type":"ContainerStarted","Data":"6afcc00c8b380f88a58b00faba1faec38911c24fe01c35eb32df8034a10342ae"}
Jul 10 18:52:50.375569 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:50.375539    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:52:50.595730 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:50.595693    2561 generic.go:356] "Generic (PLEG): container finished" podID="3560cbd5-853e-431f-b1d3-472684d59a3d" containerID="6afcc00c8b380f88a58b00faba1faec38911c24fe01c35eb32df8034a10342ae" exitCode=0
Jul 10 18:52:50.596133 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:50.595780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k" event={"ID":"3560cbd5-853e-431f-b1d3-472684d59a3d","Type":"ContainerDied","Data":"6afcc00c8b380f88a58b00faba1faec38911c24fe01c35eb32df8034a10342ae"}
Jul 10 18:52:52.368284 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:52.368244    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:52:52.609757 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:52:52.609683    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k" event={"ID":"3560cbd5-853e-431f-b1d3-472684d59a3d","Type":"ContainerStarted","Data":"d59142057a3143e7698cfd85d5c10ac39c40afa39611eed418c87e6ddd161678"}
Jul 10 18:53:01.726328 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:01.726219    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:53:11.182203 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:11.182155    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:53:12.077408 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:12.077352    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:53:15.744626 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:15.744580    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:53:19.419113 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:19.419050    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:53:22.783764 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:22.783730    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k" event={"ID":"3560cbd5-853e-431f-b1d3-472684d59a3d","Type":"ContainerStarted","Data":"5e318741b6f8de5f060238307aaf0866fee826b591909e47568892d95eda730b"}
Jul 10 18:53:22.784231 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:22.784040    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:53:22.786459 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:22.786437    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:53:22.812090 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:22.812019    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k" podStartSLOduration=2.761620298 podStartE2EDuration="37.811994133s" totalImagesPullingTime="34.380888641s" totalInitContainerRuntime="669.485194ms" isStatefulPod=false podCreationTimestamp="2026-07-10 18:52:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:53:22.810631289 +0000 UTC m=+826.284724193" watchObservedRunningTime="2026-07-10 18:53:22.811994133 +0000 UTC m=+826.286087062"
Jul 10 18:53:26.335676 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:26.335632    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:53:26.336215 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:26.335695    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:53:28.922714 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:28.922675    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:53:36.337071 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:36.337040    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:53:36.338268 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:36.338250    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:53:38.030257 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:38.030215    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"]
Jul 10 18:53:38.030676 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:38.030449    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k" podUID="3560cbd5-853e-431f-b1d3-472684d59a3d" containerName="main" containerID="cri-o://d59142057a3143e7698cfd85d5c10ac39c40afa39611eed418c87e6ddd161678" gracePeriod=30
Jul 10 18:53:38.030676 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:38.030499    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k" podUID="3560cbd5-853e-431f-b1d3-472684d59a3d" containerName="tokenizer" containerID="cri-o://5e318741b6f8de5f060238307aaf0866fee826b591909e47568892d95eda730b" gracePeriod=30
Jul 10 18:53:38.047208 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:38.047176    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:53:38.160714 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:38.160673    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:53:38.863596 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:38.863561    2561 generic.go:356] "Generic (PLEG): container finished" podID="3560cbd5-853e-431f-b1d3-472684d59a3d" containerID="d59142057a3143e7698cfd85d5c10ac39c40afa39611eed418c87e6ddd161678" exitCode=0
Jul 10 18:53:38.863778 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:38.863636    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k" event={"ID":"3560cbd5-853e-431f-b1d3-472684d59a3d","Type":"ContainerDied","Data":"d59142057a3143e7698cfd85d5c10ac39c40afa39611eed418c87e6ddd161678"}
Jul 10 18:53:39.289193 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.289167    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:53:39.422451 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.422416    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp7xn\" (UniqueName: \"kubernetes.io/projected/3560cbd5-853e-431f-b1d3-472684d59a3d-kube-api-access-qp7xn\") pod \"3560cbd5-853e-431f-b1d3-472684d59a3d\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") "
Jul 10 18:53:39.422630 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.422479    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-tokenizer-uds\") pod \"3560cbd5-853e-431f-b1d3-472684d59a3d\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") "
Jul 10 18:53:39.422630 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.422510    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-kserve-provision-location\") pod \"3560cbd5-853e-431f-b1d3-472684d59a3d\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") "
Jul 10 18:53:39.422630 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.422543    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3560cbd5-853e-431f-b1d3-472684d59a3d-tls-certs\") pod \"3560cbd5-853e-431f-b1d3-472684d59a3d\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") "
Jul 10 18:53:39.422630 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.422574    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-tokenizer-cache\") pod \"3560cbd5-853e-431f-b1d3-472684d59a3d\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") "
Jul 10 18:53:39.422865 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.422658    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-tokenizer-tmp\") pod \"3560cbd5-853e-431f-b1d3-472684d59a3d\" (UID: \"3560cbd5-853e-431f-b1d3-472684d59a3d\") "
Jul 10 18:53:39.422865 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.422794    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3560cbd5-853e-431f-b1d3-472684d59a3d" (UID: "3560cbd5-853e-431f-b1d3-472684d59a3d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:53:39.422865 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.422812    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "3560cbd5-853e-431f-b1d3-472684d59a3d" (UID: "3560cbd5-853e-431f-b1d3-472684d59a3d"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:53:39.422989 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.422976    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-tokenizer-uds\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:53:39.423033 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.422995    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-tokenizer-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:53:39.423033 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.422998    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "3560cbd5-853e-431f-b1d3-472684d59a3d" (UID: "3560cbd5-853e-431f-b1d3-472684d59a3d"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:53:39.423272 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.423253    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3560cbd5-853e-431f-b1d3-472684d59a3d" (UID: "3560cbd5-853e-431f-b1d3-472684d59a3d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:53:39.424653 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.424631    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3560cbd5-853e-431f-b1d3-472684d59a3d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3560cbd5-853e-431f-b1d3-472684d59a3d" (UID: "3560cbd5-853e-431f-b1d3-472684d59a3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:53:39.424780 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.424760    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3560cbd5-853e-431f-b1d3-472684d59a3d-kube-api-access-qp7xn" (OuterVolumeSpecName: "kube-api-access-qp7xn") pod "3560cbd5-853e-431f-b1d3-472684d59a3d" (UID: "3560cbd5-853e-431f-b1d3-472684d59a3d"). InnerVolumeSpecName "kube-api-access-qp7xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:53:39.523785 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.523756    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-tokenizer-tmp\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:53:39.523785 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.523783    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp7xn\" (UniqueName: \"kubernetes.io/projected/3560cbd5-853e-431f-b1d3-472684d59a3d-kube-api-access-qp7xn\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:53:39.523785 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.523794    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3560cbd5-853e-431f-b1d3-472684d59a3d-kserve-provision-location\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:53:39.524030 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.523804    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3560cbd5-853e-431f-b1d3-472684d59a3d-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:53:39.875395 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.875289    2561 generic.go:356] "Generic (PLEG): container finished" podID="3560cbd5-853e-431f-b1d3-472684d59a3d" containerID="5e318741b6f8de5f060238307aaf0866fee826b591909e47568892d95eda730b" exitCode=0
Jul 10 18:53:39.875395 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.875338    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k" event={"ID":"3560cbd5-853e-431f-b1d3-472684d59a3d","Type":"ContainerDied","Data":"5e318741b6f8de5f060238307aaf0866fee826b591909e47568892d95eda730b"}
Jul 10 18:53:39.875395 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.875360    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"
Jul 10 18:53:39.875664 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.875420    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k" event={"ID":"3560cbd5-853e-431f-b1d3-472684d59a3d","Type":"ContainerDied","Data":"1afe773d67b22d3acbc7537855b6ff4c9014d561c3e31c1425143741ec67d66a"}
Jul 10 18:53:39.875664 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.875449    2561 scope.go:117] "RemoveContainer" containerID="5e318741b6f8de5f060238307aaf0866fee826b591909e47568892d95eda730b"
Jul 10 18:53:39.884536 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.884514    2561 scope.go:117] "RemoveContainer" containerID="d59142057a3143e7698cfd85d5c10ac39c40afa39611eed418c87e6ddd161678"
Jul 10 18:53:39.891968 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.891943    2561 scope.go:117] "RemoveContainer" containerID="6afcc00c8b380f88a58b00faba1faec38911c24fe01c35eb32df8034a10342ae"
Jul 10 18:53:39.896974 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.896950    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"]
Jul 10 18:53:39.898629 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.898608    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k"]
Jul 10 18:53:39.900306 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.900290    2561 scope.go:117] "RemoveContainer" containerID="5e318741b6f8de5f060238307aaf0866fee826b591909e47568892d95eda730b"
Jul 10 18:53:39.900583 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:53:39.900566    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e318741b6f8de5f060238307aaf0866fee826b591909e47568892d95eda730b\": container with ID starting with 5e318741b6f8de5f060238307aaf0866fee826b591909e47568892d95eda730b not found: ID does not exist" containerID="5e318741b6f8de5f060238307aaf0866fee826b591909e47568892d95eda730b"
Jul 10 18:53:39.900627 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.900593    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e318741b6f8de5f060238307aaf0866fee826b591909e47568892d95eda730b"} err="failed to get container status \"5e318741b6f8de5f060238307aaf0866fee826b591909e47568892d95eda730b\": rpc error: code = NotFound desc = could not find container \"5e318741b6f8de5f060238307aaf0866fee826b591909e47568892d95eda730b\": container with ID starting with 5e318741b6f8de5f060238307aaf0866fee826b591909e47568892d95eda730b not found: ID does not exist"
Jul 10 18:53:39.900627 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.900611    2561 scope.go:117] "RemoveContainer" containerID="d59142057a3143e7698cfd85d5c10ac39c40afa39611eed418c87e6ddd161678"
Jul 10 18:53:39.900864 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:53:39.900846    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d59142057a3143e7698cfd85d5c10ac39c40afa39611eed418c87e6ddd161678\": container with ID starting with d59142057a3143e7698cfd85d5c10ac39c40afa39611eed418c87e6ddd161678 not found: ID does not exist" containerID="d59142057a3143e7698cfd85d5c10ac39c40afa39611eed418c87e6ddd161678"
Jul 10 18:53:39.900915 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.900871    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59142057a3143e7698cfd85d5c10ac39c40afa39611eed418c87e6ddd161678"} err="failed to get container status \"d59142057a3143e7698cfd85d5c10ac39c40afa39611eed418c87e6ddd161678\": rpc error: code = NotFound desc = could not find container \"d59142057a3143e7698cfd85d5c10ac39c40afa39611eed418c87e6ddd161678\": container with ID starting with d59142057a3143e7698cfd85d5c10ac39c40afa39611eed418c87e6ddd161678 not found: ID does not exist"
Jul 10 18:53:39.900915 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.900888    2561 scope.go:117] "RemoveContainer" containerID="6afcc00c8b380f88a58b00faba1faec38911c24fe01c35eb32df8034a10342ae"
Jul 10 18:53:39.901126 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:53:39.901111    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6afcc00c8b380f88a58b00faba1faec38911c24fe01c35eb32df8034a10342ae\": container with ID starting with 6afcc00c8b380f88a58b00faba1faec38911c24fe01c35eb32df8034a10342ae not found: ID does not exist" containerID="6afcc00c8b380f88a58b00faba1faec38911c24fe01c35eb32df8034a10342ae"
Jul 10 18:53:39.901182 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:39.901129    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6afcc00c8b380f88a58b00faba1faec38911c24fe01c35eb32df8034a10342ae"} err="failed to get container status \"6afcc00c8b380f88a58b00faba1faec38911c24fe01c35eb32df8034a10342ae\": rpc error: code = NotFound desc = could not find container \"6afcc00c8b380f88a58b00faba1faec38911c24fe01c35eb32df8034a10342ae\": container with ID starting with 6afcc00c8b380f88a58b00faba1faec38911c24fe01c35eb32df8034a10342ae not found: ID does not exist"
Jul 10 18:53:41.073001 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:41.072967    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3560cbd5-853e-431f-b1d3-472684d59a3d" path="/var/lib/kubelet/pods/3560cbd5-853e-431f-b1d3-472684d59a3d/volumes"
Jul 10 18:53:41.891158 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:41.891121    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:53:42.064411 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.064378    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"]
Jul 10 18:53:42.064758 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.064746    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3560cbd5-853e-431f-b1d3-472684d59a3d" containerName="storage-initializer"
Jul 10 18:53:42.064807 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.064759    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3560cbd5-853e-431f-b1d3-472684d59a3d" containerName="storage-initializer"
Jul 10 18:53:42.064807 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.064780    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3560cbd5-853e-431f-b1d3-472684d59a3d" containerName="main"
Jul 10 18:53:42.064807 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.064785    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3560cbd5-853e-431f-b1d3-472684d59a3d" containerName="main"
Jul 10 18:53:42.064807 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.064792    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3560cbd5-853e-431f-b1d3-472684d59a3d" containerName="tokenizer"
Jul 10 18:53:42.064807 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.064798    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3560cbd5-853e-431f-b1d3-472684d59a3d" containerName="tokenizer"
Jul 10 18:53:42.064989 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.064856    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3560cbd5-853e-431f-b1d3-472684d59a3d" containerName="main"
Jul 10 18:53:42.064989 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.064867    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3560cbd5-853e-431f-b1d3-472684d59a3d" containerName="tokenizer"
Jul 10 18:53:42.069718 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.069695    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.072234 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.072206    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 18:53:42.072364 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.072248    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jjdpb\""
Jul 10 18:53:42.072940 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.072921    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 18:53:42.072940 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.072926    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-rjkps\""
Jul 10 18:53:42.073356 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.072923    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 10 18:53:42.082098 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.082057    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"]
Jul 10 18:53:42.136796 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.136760    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:53:42.139941 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.139918    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.140069 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.139952    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hxm7\" (UniqueName: \"kubernetes.io/projected/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-kube-api-access-7hxm7\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.140069 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.139974    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.140069 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.140031    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.140220 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.140104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.140220 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.140127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.240738 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.240699    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.240945 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.240749    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.240945 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.240785    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.240945 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.240816    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hxm7\" (UniqueName: \"kubernetes.io/projected/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-kube-api-access-7hxm7\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.240945 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.240851    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.240945 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.240880    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.241202 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.241179    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.241202 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.241196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.241305 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.241284    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.241364 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.241348    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.243400 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.243378    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.249052 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.249018    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hxm7\" (UniqueName: \"kubernetes.io/projected/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-kube-api-access-7hxm7\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.379676 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.379640    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:42.515003 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.514977    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"]
Jul 10 18:53:42.517175 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:53:42.517141    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65e1f80_ddba_459b_bbc8_27f263aa9ac0.slice/crio-20738a513194c4c87e1ab9ab9989dfd1fed297dfa9ac73e0235e9406fbc33c67 WatchSource:0}: Error finding container 20738a513194c4c87e1ab9ab9989dfd1fed297dfa9ac73e0235e9406fbc33c67: Status 404 returned error can't find the container with id 20738a513194c4c87e1ab9ab9989dfd1fed297dfa9ac73e0235e9406fbc33c67
Jul 10 18:53:42.893351 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.893259    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl" event={"ID":"f65e1f80-ddba-459b-bbc8-27f263aa9ac0","Type":"ContainerStarted","Data":"4fc653616f2e04561982ecde209345c93922162ed144ac8d6808b57153c0db75"}
Jul 10 18:53:42.893351 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:42.893321    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl" event={"ID":"f65e1f80-ddba-459b-bbc8-27f263aa9ac0","Type":"ContainerStarted","Data":"20738a513194c4c87e1ab9ab9989dfd1fed297dfa9ac73e0235e9406fbc33c67"}
Jul 10 18:53:43.899830 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:43.899791    2561 generic.go:356] "Generic (PLEG): container finished" podID="f65e1f80-ddba-459b-bbc8-27f263aa9ac0" containerID="4fc653616f2e04561982ecde209345c93922162ed144ac8d6808b57153c0db75" exitCode=0
Jul 10 18:53:43.900239 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:43.899876    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl" event={"ID":"f65e1f80-ddba-459b-bbc8-27f263aa9ac0","Type":"ContainerDied","Data":"4fc653616f2e04561982ecde209345c93922162ed144ac8d6808b57153c0db75"}
Jul 10 18:53:44.907024 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:44.906992    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl" event={"ID":"f65e1f80-ddba-459b-bbc8-27f263aa9ac0","Type":"ContainerStarted","Data":"c7edab79afedc30f733829e1f8b5167451ecb50d1cdb552a21d78658795ed160"}
Jul 10 18:53:44.907538 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:44.907042    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl" event={"ID":"f65e1f80-ddba-459b-bbc8-27f263aa9ac0","Type":"ContainerStarted","Data":"f526180c8fdb43c8720c509b1cc87b33074650be54add6b8b40318ddc2b3ef79"}
Jul 10 18:53:44.907538 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:44.907221    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:44.932412 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:44.932357    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl" podStartSLOduration=2.265385691 podStartE2EDuration="2.932339721s" totalImagesPullingTime="0s" totalInitContainerRuntime="666.95403ms" isStatefulPod=false podCreationTimestamp="2026-07-10 18:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:53:44.929727822 +0000 UTC m=+848.403820720" watchObservedRunningTime="2026-07-10 18:53:44.932339721 +0000 UTC m=+848.406432621"
Jul 10 18:53:46.191450 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:46.191412    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:53:52.380569 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:52.380528    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:52.381125 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:52.380638    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:52.383151 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:52.383126    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:53:52.954206 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:53:52.954176    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:54:13.959205 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:13.959176    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:54:15.268779 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:15.268740    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"]
Jul 10 18:54:15.269203 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:15.269023    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl" podUID="f65e1f80-ddba-459b-bbc8-27f263aa9ac0" containerName="main" containerID="cri-o://f526180c8fdb43c8720c509b1cc87b33074650be54add6b8b40318ddc2b3ef79" gracePeriod=30
Jul 10 18:54:15.269203 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:15.269063    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl" podUID="f65e1f80-ddba-459b-bbc8-27f263aa9ac0" containerName="tokenizer" containerID="cri-o://c7edab79afedc30f733829e1f8b5167451ecb50d1cdb552a21d78658795ed160" gracePeriod=30
Jul 10 18:54:15.281368 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:15.281334    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:54:15.381812 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:15.381782    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:54:16.066426 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.066390    2561 generic.go:356] "Generic (PLEG): container finished" podID="f65e1f80-ddba-459b-bbc8-27f263aa9ac0" containerID="f526180c8fdb43c8720c509b1cc87b33074650be54add6b8b40318ddc2b3ef79" exitCode=0
Jul 10 18:54:16.066645 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.066439    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl" event={"ID":"f65e1f80-ddba-459b-bbc8-27f263aa9ac0","Type":"ContainerDied","Data":"f526180c8fdb43c8720c509b1cc87b33074650be54add6b8b40318ddc2b3ef79"}
Jul 10 18:54:16.638088 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.638046    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:54:16.740647 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.740568    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tls-certs\") pod \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") "
Jul 10 18:54:16.740647 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.740631    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tokenizer-cache\") pod \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") "
Jul 10 18:54:16.740870 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.740699    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tokenizer-tmp\") pod \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") "
Jul 10 18:54:16.740870 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.740762    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tokenizer-uds\") pod \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") "
Jul 10 18:54:16.740870 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.740814    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hxm7\" (UniqueName: \"kubernetes.io/projected/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-kube-api-access-7hxm7\") pod \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") "
Jul 10 18:54:16.740870 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.740843    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-kserve-provision-location\") pod \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\" (UID: \"f65e1f80-ddba-459b-bbc8-27f263aa9ac0\") "
Jul 10 18:54:16.741105 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.740901    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "f65e1f80-ddba-459b-bbc8-27f263aa9ac0" (UID: "f65e1f80-ddba-459b-bbc8-27f263aa9ac0"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:54:16.741105 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.741016    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f65e1f80-ddba-459b-bbc8-27f263aa9ac0" (UID: "f65e1f80-ddba-459b-bbc8-27f263aa9ac0"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:54:16.741105 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.741069    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "f65e1f80-ddba-459b-bbc8-27f263aa9ac0" (UID: "f65e1f80-ddba-459b-bbc8-27f263aa9ac0"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:54:16.741284 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.741267    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tokenizer-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:54:16.741345 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.741291    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tokenizer-tmp\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:54:16.741345 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.741305    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tokenizer-uds\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:54:16.741697 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.741670    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f65e1f80-ddba-459b-bbc8-27f263aa9ac0" (UID: "f65e1f80-ddba-459b-bbc8-27f263aa9ac0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:54:16.742778 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.742757    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f65e1f80-ddba-459b-bbc8-27f263aa9ac0" (UID: "f65e1f80-ddba-459b-bbc8-27f263aa9ac0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:54:16.742921 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.742900    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-kube-api-access-7hxm7" (OuterVolumeSpecName: "kube-api-access-7hxm7") pod "f65e1f80-ddba-459b-bbc8-27f263aa9ac0" (UID: "f65e1f80-ddba-459b-bbc8-27f263aa9ac0"). InnerVolumeSpecName "kube-api-access-7hxm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:54:16.842449 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.842409    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hxm7\" (UniqueName: \"kubernetes.io/projected/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-kube-api-access-7hxm7\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:54:16.842449 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.842442    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-kserve-provision-location\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:54:16.842449 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:16.842452    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f65e1f80-ddba-459b-bbc8-27f263aa9ac0-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:54:17.074374 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:17.074281    2561 generic.go:356] "Generic (PLEG): container finished" podID="f65e1f80-ddba-459b-bbc8-27f263aa9ac0" containerID="c7edab79afedc30f733829e1f8b5167451ecb50d1cdb552a21d78658795ed160" exitCode=0
Jul 10 18:54:17.074374 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:17.074336    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"
Jul 10 18:54:17.074374 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:17.074362    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl" event={"ID":"f65e1f80-ddba-459b-bbc8-27f263aa9ac0","Type":"ContainerDied","Data":"c7edab79afedc30f733829e1f8b5167451ecb50d1cdb552a21d78658795ed160"}
Jul 10 18:54:17.074640 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:17.074412    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl" event={"ID":"f65e1f80-ddba-459b-bbc8-27f263aa9ac0","Type":"ContainerDied","Data":"20738a513194c4c87e1ab9ab9989dfd1fed297dfa9ac73e0235e9406fbc33c67"}
Jul 10 18:54:17.074640 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:17.074436    2561 scope.go:117] "RemoveContainer" containerID="c7edab79afedc30f733829e1f8b5167451ecb50d1cdb552a21d78658795ed160"
Jul 10 18:54:17.083645 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:17.083629    2561 scope.go:117] "RemoveContainer" containerID="f526180c8fdb43c8720c509b1cc87b33074650be54add6b8b40318ddc2b3ef79"
Jul 10 18:54:17.091346 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:17.091329    2561 scope.go:117] "RemoveContainer" containerID="4fc653616f2e04561982ecde209345c93922162ed144ac8d6808b57153c0db75"
Jul 10 18:54:17.099419 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:17.099391    2561 scope.go:117] "RemoveContainer" containerID="c7edab79afedc30f733829e1f8b5167451ecb50d1cdb552a21d78658795ed160"
Jul 10 18:54:17.099731 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:54:17.099701    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7edab79afedc30f733829e1f8b5167451ecb50d1cdb552a21d78658795ed160\": container with ID starting with c7edab79afedc30f733829e1f8b5167451ecb50d1cdb552a21d78658795ed160 not found: ID does not exist" containerID="c7edab79afedc30f733829e1f8b5167451ecb50d1cdb552a21d78658795ed160"
Jul 10 18:54:17.099786 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:17.099740    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7edab79afedc30f733829e1f8b5167451ecb50d1cdb552a21d78658795ed160"} err="failed to get container status \"c7edab79afedc30f733829e1f8b5167451ecb50d1cdb552a21d78658795ed160\": rpc error: code = NotFound desc = could not find container \"c7edab79afedc30f733829e1f8b5167451ecb50d1cdb552a21d78658795ed160\": container with ID starting with c7edab79afedc30f733829e1f8b5167451ecb50d1cdb552a21d78658795ed160 not found: ID does not exist"
Jul 10 18:54:17.099786 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:17.099759    2561 scope.go:117] "RemoveContainer" containerID="f526180c8fdb43c8720c509b1cc87b33074650be54add6b8b40318ddc2b3ef79"
Jul 10 18:54:17.100026 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:54:17.100008    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f526180c8fdb43c8720c509b1cc87b33074650be54add6b8b40318ddc2b3ef79\": container with ID starting with f526180c8fdb43c8720c509b1cc87b33074650be54add6b8b40318ddc2b3ef79 not found: ID does not exist" containerID="f526180c8fdb43c8720c509b1cc87b33074650be54add6b8b40318ddc2b3ef79"
Jul 10 18:54:17.100084 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:17.100035    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f526180c8fdb43c8720c509b1cc87b33074650be54add6b8b40318ddc2b3ef79"} err="failed to get container status \"f526180c8fdb43c8720c509b1cc87b33074650be54add6b8b40318ddc2b3ef79\": rpc error: code = NotFound desc = could not find container \"f526180c8fdb43c8720c509b1cc87b33074650be54add6b8b40318ddc2b3ef79\": container with ID starting with f526180c8fdb43c8720c509b1cc87b33074650be54add6b8b40318ddc2b3ef79 not found: ID does not exist"
Jul 10 18:54:17.100084 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:17.100052    2561 scope.go:117] "RemoveContainer" containerID="4fc653616f2e04561982ecde209345c93922162ed144ac8d6808b57153c0db75"
Jul 10 18:54:17.100295 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:54:17.100277    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fc653616f2e04561982ecde209345c93922162ed144ac8d6808b57153c0db75\": container with ID starting with 4fc653616f2e04561982ecde209345c93922162ed144ac8d6808b57153c0db75 not found: ID does not exist" containerID="4fc653616f2e04561982ecde209345c93922162ed144ac8d6808b57153c0db75"
Jul 10 18:54:17.100342 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:17.100290    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"]
Jul 10 18:54:17.100342 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:17.100299    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fc653616f2e04561982ecde209345c93922162ed144ac8d6808b57153c0db75"} err="failed to get container status \"4fc653616f2e04561982ecde209345c93922162ed144ac8d6808b57153c0db75\": rpc error: code = NotFound desc = could not find container \"4fc653616f2e04561982ecde209345c93922162ed144ac8d6808b57153c0db75\": container with ID starting with 4fc653616f2e04561982ecde209345c93922162ed144ac8d6808b57153c0db75 not found: ID does not exist"
Jul 10 18:54:17.104044 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:17.104022    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl"]
Jul 10 18:54:19.075168 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:19.075133    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f65e1f80-ddba-459b-bbc8-27f263aa9ac0" path="/var/lib/kubelet/pods/f65e1f80-ddba-459b-bbc8-27f263aa9ac0/volumes"
Jul 10 18:54:28.066697 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.066649    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:54:28.220731 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.220700    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"]
Jul 10 18:54:28.221070 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.221058    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f65e1f80-ddba-459b-bbc8-27f263aa9ac0" containerName="main"
Jul 10 18:54:28.221070 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.221084    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f65e1f80-ddba-459b-bbc8-27f263aa9ac0" containerName="main"
Jul 10 18:54:28.221174 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.221099    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f65e1f80-ddba-459b-bbc8-27f263aa9ac0" containerName="tokenizer"
Jul 10 18:54:28.221174 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.221105    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f65e1f80-ddba-459b-bbc8-27f263aa9ac0" containerName="tokenizer"
Jul 10 18:54:28.221174 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.221119    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f65e1f80-ddba-459b-bbc8-27f263aa9ac0" containerName="storage-initializer"
Jul 10 18:54:28.221174 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.221125    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f65e1f80-ddba-459b-bbc8-27f263aa9ac0" containerName="storage-initializer"
Jul 10 18:54:28.221292 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.221187    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f65e1f80-ddba-459b-bbc8-27f263aa9ac0" containerName="tokenizer"
Jul 10 18:54:28.221292 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.221201    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f65e1f80-ddba-459b-bbc8-27f263aa9ac0" containerName="main"
Jul 10 18:54:28.226206 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.226177    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.234704 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.234684    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 10 18:54:28.234704 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.234694    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 18:54:28.234876 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.234706    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 18:54:28.235677 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.235657    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jjdpb\""
Jul 10 18:54:28.235776 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.235726    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-kq4vp\""
Jul 10 18:54:28.247597 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.247572    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"]
Jul 10 18:54:28.269109 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.269059    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:54:28.343771 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.343682    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.343771 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.343739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.343771 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.343766    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb4lm\" (UniqueName: \"kubernetes.io/projected/f55978c5-63ed-48b5-89cd-e4301cc65f19-kube-api-access-lb4lm\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.344023 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.343822    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f55978c5-63ed-48b5-89cd-e4301cc65f19-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.344023 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.343863    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.344023 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.343927    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.445163 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.445129    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.445345 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.445178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.445345 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.445224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.445345 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.445245    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb4lm\" (UniqueName: \"kubernetes.io/projected/f55978c5-63ed-48b5-89cd-e4301cc65f19-kube-api-access-lb4lm\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.445345 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.445278    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f55978c5-63ed-48b5-89cd-e4301cc65f19-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.445345 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.445311    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.445613 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.445586    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.445613 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.445599    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.445748 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.445658    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.445748 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.445673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.447744 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.447719    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f55978c5-63ed-48b5-89cd-e4301cc65f19-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.454628 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.454603    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb4lm\" (UniqueName: \"kubernetes.io/projected/f55978c5-63ed-48b5-89cd-e4301cc65f19-kube-api-access-lb4lm\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.561838 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.561793    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:28.759068 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:28.759037    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"]
Jul 10 18:54:28.760640 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:54:28.760614    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55978c5_63ed_48b5_89cd_e4301cc65f19.slice/crio-7b3f0da4caedb823ecdcee1a47c4ce054eb0d7c48d3cbbb1cb281218448164f8 WatchSource:0}: Error finding container 7b3f0da4caedb823ecdcee1a47c4ce054eb0d7c48d3cbbb1cb281218448164f8: Status 404 returned error can't find the container with id 7b3f0da4caedb823ecdcee1a47c4ce054eb0d7c48d3cbbb1cb281218448164f8
Jul 10 18:54:29.131808 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:29.131770    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77" event={"ID":"f55978c5-63ed-48b5-89cd-e4301cc65f19","Type":"ContainerStarted","Data":"70725a5b419c7ac07509a8d2a44052352da86b0168e6e7bb1fd3ae73a1467d48"}
Jul 10 18:54:29.132205 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:29.131827    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77" event={"ID":"f55978c5-63ed-48b5-89cd-e4301cc65f19","Type":"ContainerStarted","Data":"7b3f0da4caedb823ecdcee1a47c4ce054eb0d7c48d3cbbb1cb281218448164f8"}
Jul 10 18:54:30.137460 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:30.137425    2561 generic.go:356] "Generic (PLEG): container finished" podID="f55978c5-63ed-48b5-89cd-e4301cc65f19" containerID="70725a5b419c7ac07509a8d2a44052352da86b0168e6e7bb1fd3ae73a1467d48" exitCode=0
Jul 10 18:54:30.137893 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:30.137506    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77" event={"ID":"f55978c5-63ed-48b5-89cd-e4301cc65f19","Type":"ContainerDied","Data":"70725a5b419c7ac07509a8d2a44052352da86b0168e6e7bb1fd3ae73a1467d48"}
Jul 10 18:54:31.144331 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:31.144293    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77" event={"ID":"f55978c5-63ed-48b5-89cd-e4301cc65f19","Type":"ContainerStarted","Data":"1b888f48d2a7de3083f354f12146042a8f0e4b3b0e5933e7f401c39965333b14"}
Jul 10 18:54:31.144711 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:31.144345    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77" event={"ID":"f55978c5-63ed-48b5-89cd-e4301cc65f19","Type":"ContainerStarted","Data":"e5f4514f93d281b838cc5d39c989ff9344e43f58307f52003244a936011c23a5"}
Jul 10 18:54:31.144711 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:31.144449    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:31.182657 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:31.182609    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77" podStartSLOduration=2.421220682 podStartE2EDuration="3.182594112s" totalImagesPullingTime="0s" totalInitContainerRuntime="761.37343ms" isStatefulPod=false podCreationTimestamp="2026-07-10 18:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:54:31.181237632 +0000 UTC m=+894.655330516" watchObservedRunningTime="2026-07-10 18:54:31.182594112 +0000 UTC m=+894.656686997"
Jul 10 18:54:34.844778 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:34.844739    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:54:37.028799 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:37.028767    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 18:54:37.035156 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:37.035123    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 18:54:37.037212 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:37.037187    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 18:54:37.042771 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:37.042750    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 18:54:38.561924 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:38.561891    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:38.562420 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:38.562099    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:38.564847 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:38.564825    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:54:39.184636 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:54:39.184605    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:55:01.195875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:01.195845    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:55:02.712758 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:02.712717    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"]
Jul 10 18:55:02.713221 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:02.713003    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77" podUID="f55978c5-63ed-48b5-89cd-e4301cc65f19" containerName="main" containerID="cri-o://e5f4514f93d281b838cc5d39c989ff9344e43f58307f52003244a936011c23a5" gracePeriod=30
Jul 10 18:55:02.713221 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:02.713044    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77" podUID="f55978c5-63ed-48b5-89cd-e4301cc65f19" containerName="tokenizer" containerID="cri-o://1b888f48d2a7de3083f354f12146042a8f0e4b3b0e5933e7f401c39965333b14" gracePeriod=30
Jul 10 18:55:02.731624 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:02.731587    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:55:02.796620 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:02.796580    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:55:03.298668 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:03.298634    2561 generic.go:356] "Generic (PLEG): container finished" podID="f55978c5-63ed-48b5-89cd-e4301cc65f19" containerID="e5f4514f93d281b838cc5d39c989ff9344e43f58307f52003244a936011c23a5" exitCode=0
Jul 10 18:55:03.298854 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:03.298713    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77" event={"ID":"f55978c5-63ed-48b5-89cd-e4301cc65f19","Type":"ContainerDied","Data":"e5f4514f93d281b838cc5d39c989ff9344e43f58307f52003244a936011c23a5"}
Jul 10 18:55:04.065299 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.065274    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:55:04.163614 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.163582    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb4lm\" (UniqueName: \"kubernetes.io/projected/f55978c5-63ed-48b5-89cd-e4301cc65f19-kube-api-access-lb4lm\") pod \"f55978c5-63ed-48b5-89cd-e4301cc65f19\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") "
Jul 10 18:55:04.163789 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.163643    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f55978c5-63ed-48b5-89cd-e4301cc65f19-tls-certs\") pod \"f55978c5-63ed-48b5-89cd-e4301cc65f19\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") "
Jul 10 18:55:04.163789 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.163668    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-tokenizer-uds\") pod \"f55978c5-63ed-48b5-89cd-e4301cc65f19\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") "
Jul 10 18:55:04.163789 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.163693    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-kserve-provision-location\") pod \"f55978c5-63ed-48b5-89cd-e4301cc65f19\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") "
Jul 10 18:55:04.163789 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.163721    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-tokenizer-cache\") pod \"f55978c5-63ed-48b5-89cd-e4301cc65f19\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") "
Jul 10 18:55:04.163789 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.163738    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-tokenizer-tmp\") pod \"f55978c5-63ed-48b5-89cd-e4301cc65f19\" (UID: \"f55978c5-63ed-48b5-89cd-e4301cc65f19\") "
Jul 10 18:55:04.164089 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.163967    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f55978c5-63ed-48b5-89cd-e4301cc65f19" (UID: "f55978c5-63ed-48b5-89cd-e4301cc65f19"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:04.164155 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.164064    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "f55978c5-63ed-48b5-89cd-e4301cc65f19" (UID: "f55978c5-63ed-48b5-89cd-e4301cc65f19"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:04.164197 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.164160    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "f55978c5-63ed-48b5-89cd-e4301cc65f19" (UID: "f55978c5-63ed-48b5-89cd-e4301cc65f19"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:04.164512 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.164484    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f55978c5-63ed-48b5-89cd-e4301cc65f19" (UID: "f55978c5-63ed-48b5-89cd-e4301cc65f19"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:04.165948 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.165928    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55978c5-63ed-48b5-89cd-e4301cc65f19-kube-api-access-lb4lm" (OuterVolumeSpecName: "kube-api-access-lb4lm") pod "f55978c5-63ed-48b5-89cd-e4301cc65f19" (UID: "f55978c5-63ed-48b5-89cd-e4301cc65f19"). InnerVolumeSpecName "kube-api-access-lb4lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:55:04.166031 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.165957    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f55978c5-63ed-48b5-89cd-e4301cc65f19-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f55978c5-63ed-48b5-89cd-e4301cc65f19" (UID: "f55978c5-63ed-48b5-89cd-e4301cc65f19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:55:04.264868 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.264823    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb4lm\" (UniqueName: \"kubernetes.io/projected/f55978c5-63ed-48b5-89cd-e4301cc65f19-kube-api-access-lb4lm\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:04.264868 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.264856    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f55978c5-63ed-48b5-89cd-e4301cc65f19-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:04.264868 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.264866    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-tokenizer-uds\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:04.264868 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.264875    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-kserve-provision-location\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:04.265173 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.264883    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-tokenizer-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:04.265173 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.264892    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f55978c5-63ed-48b5-89cd-e4301cc65f19-tokenizer-tmp\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:04.305687 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.305652    2561 generic.go:356] "Generic (PLEG): container finished" podID="f55978c5-63ed-48b5-89cd-e4301cc65f19" containerID="1b888f48d2a7de3083f354f12146042a8f0e4b3b0e5933e7f401c39965333b14" exitCode=0
Jul 10 18:55:04.305848 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.305715    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"
Jul 10 18:55:04.305848 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.305719    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77" event={"ID":"f55978c5-63ed-48b5-89cd-e4301cc65f19","Type":"ContainerDied","Data":"1b888f48d2a7de3083f354f12146042a8f0e4b3b0e5933e7f401c39965333b14"}
Jul 10 18:55:04.305848 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.305767    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77" event={"ID":"f55978c5-63ed-48b5-89cd-e4301cc65f19","Type":"ContainerDied","Data":"7b3f0da4caedb823ecdcee1a47c4ce054eb0d7c48d3cbbb1cb281218448164f8"}
Jul 10 18:55:04.305848 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.305793    2561 scope.go:117] "RemoveContainer" containerID="1b888f48d2a7de3083f354f12146042a8f0e4b3b0e5933e7f401c39965333b14"
Jul 10 18:55:04.315525 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.315506    2561 scope.go:117] "RemoveContainer" containerID="e5f4514f93d281b838cc5d39c989ff9344e43f58307f52003244a936011c23a5"
Jul 10 18:55:04.325774 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.325750    2561 scope.go:117] "RemoveContainer" containerID="70725a5b419c7ac07509a8d2a44052352da86b0168e6e7bb1fd3ae73a1467d48"
Jul 10 18:55:04.328814 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.328786    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"]
Jul 10 18:55:04.332888 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.332863    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77"]
Jul 10 18:55:04.334475 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.334454    2561 scope.go:117] "RemoveContainer" containerID="1b888f48d2a7de3083f354f12146042a8f0e4b3b0e5933e7f401c39965333b14"
Jul 10 18:55:04.334796 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:55:04.334777    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b888f48d2a7de3083f354f12146042a8f0e4b3b0e5933e7f401c39965333b14\": container with ID starting with 1b888f48d2a7de3083f354f12146042a8f0e4b3b0e5933e7f401c39965333b14 not found: ID does not exist" containerID="1b888f48d2a7de3083f354f12146042a8f0e4b3b0e5933e7f401c39965333b14"
Jul 10 18:55:04.334881 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.334810    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b888f48d2a7de3083f354f12146042a8f0e4b3b0e5933e7f401c39965333b14"} err="failed to get container status \"1b888f48d2a7de3083f354f12146042a8f0e4b3b0e5933e7f401c39965333b14\": rpc error: code = NotFound desc = could not find container \"1b888f48d2a7de3083f354f12146042a8f0e4b3b0e5933e7f401c39965333b14\": container with ID starting with 1b888f48d2a7de3083f354f12146042a8f0e4b3b0e5933e7f401c39965333b14 not found: ID does not exist"
Jul 10 18:55:04.334881 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.334839    2561 scope.go:117] "RemoveContainer" containerID="e5f4514f93d281b838cc5d39c989ff9344e43f58307f52003244a936011c23a5"
Jul 10 18:55:04.335101 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:55:04.335067    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5f4514f93d281b838cc5d39c989ff9344e43f58307f52003244a936011c23a5\": container with ID starting with e5f4514f93d281b838cc5d39c989ff9344e43f58307f52003244a936011c23a5 not found: ID does not exist" containerID="e5f4514f93d281b838cc5d39c989ff9344e43f58307f52003244a936011c23a5"
Jul 10 18:55:04.335158 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.335110    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f4514f93d281b838cc5d39c989ff9344e43f58307f52003244a936011c23a5"} err="failed to get container status \"e5f4514f93d281b838cc5d39c989ff9344e43f58307f52003244a936011c23a5\": rpc error: code = NotFound desc = could not find container \"e5f4514f93d281b838cc5d39c989ff9344e43f58307f52003244a936011c23a5\": container with ID starting with e5f4514f93d281b838cc5d39c989ff9344e43f58307f52003244a936011c23a5 not found: ID does not exist"
Jul 10 18:55:04.335158 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.335132    2561 scope.go:117] "RemoveContainer" containerID="70725a5b419c7ac07509a8d2a44052352da86b0168e6e7bb1fd3ae73a1467d48"
Jul 10 18:55:04.335374 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:55:04.335357    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70725a5b419c7ac07509a8d2a44052352da86b0168e6e7bb1fd3ae73a1467d48\": container with ID starting with 70725a5b419c7ac07509a8d2a44052352da86b0168e6e7bb1fd3ae73a1467d48 not found: ID does not exist" containerID="70725a5b419c7ac07509a8d2a44052352da86b0168e6e7bb1fd3ae73a1467d48"
Jul 10 18:55:04.335430 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:04.335382    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70725a5b419c7ac07509a8d2a44052352da86b0168e6e7bb1fd3ae73a1467d48"} err="failed to get container status \"70725a5b419c7ac07509a8d2a44052352da86b0168e6e7bb1fd3ae73a1467d48\": rpc error: code = NotFound desc = could not find container \"70725a5b419c7ac07509a8d2a44052352da86b0168e6e7bb1fd3ae73a1467d48\": container with ID starting with 70725a5b419c7ac07509a8d2a44052352da86b0168e6e7bb1fd3ae73a1467d48 not found: ID does not exist"
Jul 10 18:55:05.072982 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:05.072947    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f55978c5-63ed-48b5-89cd-e4301cc65f19" path="/var/lib/kubelet/pods/f55978c5-63ed-48b5-89cd-e4301cc65f19/volumes"
Jul 10 18:55:19.929317 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:19.929280    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"]
Jul 10 18:55:19.929680 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:19.929643    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55978c5-63ed-48b5-89cd-e4301cc65f19" containerName="storage-initializer"
Jul 10 18:55:19.929680 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:19.929654    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55978c5-63ed-48b5-89cd-e4301cc65f19" containerName="storage-initializer"
Jul 10 18:55:19.929680 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:19.929667    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55978c5-63ed-48b5-89cd-e4301cc65f19" containerName="main"
Jul 10 18:55:19.929680 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:19.929673    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55978c5-63ed-48b5-89cd-e4301cc65f19" containerName="main"
Jul 10 18:55:19.929813 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:19.929687    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55978c5-63ed-48b5-89cd-e4301cc65f19" containerName="tokenizer"
Jul 10 18:55:19.929813 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:19.929693    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55978c5-63ed-48b5-89cd-e4301cc65f19" containerName="tokenizer"
Jul 10 18:55:19.929813 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:19.929747    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f55978c5-63ed-48b5-89cd-e4301cc65f19" containerName="tokenizer"
Jul 10 18:55:19.929813 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:19.929754    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f55978c5-63ed-48b5-89cd-e4301cc65f19" containerName="main"
Jul 10 18:55:19.934660 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:19.934637    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:19.937216 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:19.937190    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 18:55:19.938113 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:19.938092    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 18:55:19.938113 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:19.938102    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jjdpb\""
Jul 10 18:55:19.938264 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:19.938098    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 10 18:55:19.943997 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:19.943967    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"]
Jul 10 18:55:19.961347 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:19.961314    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:55:20.110096 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.110050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-model-cache\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.110294 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.110127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c9aabd-65b8-467e-b35b-5b3210750fb0-tls-certs\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.110294 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.110165    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.110294 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.110250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzxdv\" (UniqueName: \"kubernetes.io/projected/54c9aabd-65b8-467e-b35b-5b3210750fb0-kube-api-access-dzxdv\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.110428 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.110295    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-dshm\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.110428 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.110317    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.110428 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.110368    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-home\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.181859 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.181773    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"]
Jul 10 18:55:20.185428 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.185407    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.188281 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.188261    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-pn6fp\""
Jul 10 18:55:20.201143 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.201117    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"]
Jul 10 18:55:20.211068 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.211031    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-dshm\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.211068 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.211066    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.211347 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.211130    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-home\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.211347 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.211188    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-model-cache\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.211347 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.211218    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c9aabd-65b8-467e-b35b-5b3210750fb0-tls-certs\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.211347 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.211264    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.211347 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.211322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzxdv\" (UniqueName: \"kubernetes.io/projected/54c9aabd-65b8-467e-b35b-5b3210750fb0-kube-api-access-dzxdv\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.211674 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.211575    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-home\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.211823 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.211760    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-model-cache\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.211823 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.211794    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.211951 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.211904    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.212649 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.212623    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:55:20.214288 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.214264    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-dshm\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.214387 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.214348    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c9aabd-65b8-467e-b35b-5b3210750fb0-tls-certs\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.219694 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.219672    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzxdv\" (UniqueName: \"kubernetes.io/projected/54c9aabd-65b8-467e-b35b-5b3210750fb0-kube-api-access-dzxdv\") pod \"precise-prefix-cache-test-kserve-6657b58549-dfrnm\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.246897 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.246861    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:20.312525 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.312482    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d853843-1cf6-4292-8c25-321aaaea25d3-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.312841 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.312808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl48c\" (UniqueName: \"kubernetes.io/projected/9d853843-1cf6-4292-8c25-321aaaea25d3-kube-api-access-tl48c\") pod \"precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.312962 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.312923    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.313038 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.312996    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.313135 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.313070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.313236 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.313219    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.387418 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.387387    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"]
Jul 10 18:55:20.389953 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:55:20.389919    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c9aabd_65b8_467e_b35b_5b3210750fb0.slice/crio-adb08dc8bbd1d75a0e2efe733a0ee5bdf085c147d040934e83a1878c10f4c771 WatchSource:0}: Error finding container adb08dc8bbd1d75a0e2efe733a0ee5bdf085c147d040934e83a1878c10f4c771: Status 404 returned error can't find the container with id adb08dc8bbd1d75a0e2efe733a0ee5bdf085c147d040934e83a1878c10f4c771
Jul 10 18:55:20.413806 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.413777    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.413931 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.413838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d853843-1cf6-4292-8c25-321aaaea25d3-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.413931 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.413882    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl48c\" (UniqueName: \"kubernetes.io/projected/9d853843-1cf6-4292-8c25-321aaaea25d3-kube-api-access-tl48c\") pod \"precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.413931 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.413920    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.414118 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.413952    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.414118 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.413989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.414232 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.414209    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.414327 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.414301    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.414387 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.414369    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.414438 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.414375    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.416317 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.416299    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d853843-1cf6-4292-8c25-321aaaea25d3-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.421766 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.421746    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl48c\" (UniqueName: \"kubernetes.io/projected/9d853843-1cf6-4292-8c25-321aaaea25d3-kube-api-access-tl48c\") pod \"precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.494940 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.494904    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:20.640264 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:20.640231    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"]
Jul 10 18:55:20.642861 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:55:20.642831    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d853843_1cf6_4292_8c25_321aaaea25d3.slice/crio-2ec8fa0cf33cf8baa1251def167a31efaf7d821ab7da72f27e5bf99de8b0cb37 WatchSource:0}: Error finding container 2ec8fa0cf33cf8baa1251def167a31efaf7d821ab7da72f27e5bf99de8b0cb37: Status 404 returned error can't find the container with id 2ec8fa0cf33cf8baa1251def167a31efaf7d821ab7da72f27e5bf99de8b0cb37
Jul 10 18:55:21.391317 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:21.391264    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm" event={"ID":"54c9aabd-65b8-467e-b35b-5b3210750fb0","Type":"ContainerStarted","Data":"4df42683ac717212b409400dc424500f48acfcd6bb912bd1e18793795c39b8d5"}
Jul 10 18:55:21.391956 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:21.391345    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm" event={"ID":"54c9aabd-65b8-467e-b35b-5b3210750fb0","Type":"ContainerStarted","Data":"adb08dc8bbd1d75a0e2efe733a0ee5bdf085c147d040934e83a1878c10f4c771"}
Jul 10 18:55:21.393132 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:21.393103    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589" event={"ID":"9d853843-1cf6-4292-8c25-321aaaea25d3","Type":"ContainerStarted","Data":"77ebc48b068846ab77606c1ca191cceebc8580658f3cbe8d48d6c0de25659592"}
Jul 10 18:55:21.393254 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:21.393164    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589" event={"ID":"9d853843-1cf6-4292-8c25-321aaaea25d3","Type":"ContainerStarted","Data":"2ec8fa0cf33cf8baa1251def167a31efaf7d821ab7da72f27e5bf99de8b0cb37"}
Jul 10 18:55:22.401424 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:22.401386    2561 generic.go:356] "Generic (PLEG): container finished" podID="9d853843-1cf6-4292-8c25-321aaaea25d3" containerID="77ebc48b068846ab77606c1ca191cceebc8580658f3cbe8d48d6c0de25659592" exitCode=0
Jul 10 18:55:22.401826 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:22.401482    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589" event={"ID":"9d853843-1cf6-4292-8c25-321aaaea25d3","Type":"ContainerDied","Data":"77ebc48b068846ab77606c1ca191cceebc8580658f3cbe8d48d6c0de25659592"}
Jul 10 18:55:23.411467 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:23.411423    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589" event={"ID":"9d853843-1cf6-4292-8c25-321aaaea25d3","Type":"ContainerStarted","Data":"e17bfe10bb499535ab130f0982f9564eaa6a9a79b7bb6f9436aa695c76380abc"}
Jul 10 18:55:23.411912 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:23.411492    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589" event={"ID":"9d853843-1cf6-4292-8c25-321aaaea25d3","Type":"ContainerStarted","Data":"7d53e1fb283f565167a91177a3e2f280b3694e74efba3e1fdd1d05482331623c"}
Jul 10 18:55:23.411912 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:23.411543    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:23.434866 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:23.434782    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589" podStartSLOduration=2.736403873 podStartE2EDuration="3.434763288s" totalImagesPullingTime="0s" totalInitContainerRuntime="698.359415ms" isStatefulPod=false podCreationTimestamp="2026-07-10 18:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:55:23.433256766 +0000 UTC m=+946.907349652" watchObservedRunningTime="2026-07-10 18:55:23.434763288 +0000 UTC m=+946.908856174"
Jul 10 18:55:25.423581 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:25.423547    2561 generic.go:356] "Generic (PLEG): container finished" podID="54c9aabd-65b8-467e-b35b-5b3210750fb0" containerID="4df42683ac717212b409400dc424500f48acfcd6bb912bd1e18793795c39b8d5" exitCode=0
Jul 10 18:55:25.424004 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:25.423634    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm" event={"ID":"54c9aabd-65b8-467e-b35b-5b3210750fb0","Type":"ContainerDied","Data":"4df42683ac717212b409400dc424500f48acfcd6bb912bd1e18793795c39b8d5"}
Jul 10 18:55:27.435797 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:27.435753    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm" event={"ID":"54c9aabd-65b8-467e-b35b-5b3210750fb0","Type":"ContainerStarted","Data":"c404790da86ee6c6dd0db3e96937d5c8adbcbf9beb1acd86b334dfa8677d7438"}
Jul 10 18:55:27.458552 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:27.458497    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm" podStartSLOduration=2.7621114220000003 podStartE2EDuration="8.458480816s" totalImagesPullingTime="1.189040672s" totalInitContainerRuntime="4.507328722s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:55:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:55:27.456600629 +0000 UTC m=+950.930693540" watchObservedRunningTime="2026-07-10 18:55:27.458480816 +0000 UTC m=+950.932573753"
Jul 10 18:55:28.333423 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:28.333382    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:55:30.247011 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:30.246959    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:30.247469 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:30.247032    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:30.259622 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:30.259588    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:30.462965 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:30.462931    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:30.495251 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:30.495221    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:30.495421 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:30.495299    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:30.497181 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:55:30.497115    2561 logging.go:55] [core] [Channel #53 SubChannel #54]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.37:9003", ServerName: "10.132.0.37:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.37:9003: connect: connection refused"
Jul 10 18:55:30.497908 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:30.497883    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:31.457844 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:31.457810    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:31.495357 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:31.495311    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589" podUID="9d853843-1cf6-4292-8c25-321aaaea25d3" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.37:9003\" within 1s: context deadline exceeded"
Jul 10 18:55:40.496172 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:55:40.496136    2561 logging.go:55] [core] [Channel #55 SubChannel #56]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.37:9003", ServerName: "10.132.0.37:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.37:9003: connect: connection refused"
Jul 10 18:55:41.495894 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:41.495849    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589" podUID="9d853843-1cf6-4292-8c25-321aaaea25d3" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.37:9003\" within 1s: context deadline exceeded"
Jul 10 18:55:52.463647 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:52.463617    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:53.362218 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.361901    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"]
Jul 10 18:55:53.362218 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.362199    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589" podUID="9d853843-1cf6-4292-8c25-321aaaea25d3" containerName="main" containerID="cri-o://7d53e1fb283f565167a91177a3e2f280b3694e74efba3e1fdd1d05482331623c" gracePeriod=30
Jul 10 18:55:53.362573 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.362544    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589" podUID="9d853843-1cf6-4292-8c25-321aaaea25d3" containerName="tokenizer" containerID="cri-o://e17bfe10bb499535ab130f0982f9564eaa6a9a79b7bb6f9436aa695c76380abc" gracePeriod=30
Jul 10 18:55:53.372023 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.371982    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"]
Jul 10 18:55:53.372332 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.372279    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm" podUID="54c9aabd-65b8-467e-b35b-5b3210750fb0" containerName="main" containerID="cri-o://c404790da86ee6c6dd0db3e96937d5c8adbcbf9beb1acd86b334dfa8677d7438" gracePeriod=30
Jul 10 18:55:53.379601 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.379579    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:55:53.437758 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.437723    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:55:53.569447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.569209    2561 generic.go:356] "Generic (PLEG): container finished" podID="9d853843-1cf6-4292-8c25-321aaaea25d3" containerID="7d53e1fb283f565167a91177a3e2f280b3694e74efba3e1fdd1d05482331623c" exitCode=0
Jul 10 18:55:53.569447 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.569373    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589" event={"ID":"9d853843-1cf6-4292-8c25-321aaaea25d3","Type":"ContainerDied","Data":"7d53e1fb283f565167a91177a3e2f280b3694e74efba3e1fdd1d05482331623c"}
Jul 10 18:55:53.572928 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.572693    2561 generic.go:356] "Generic (PLEG): container finished" podID="54c9aabd-65b8-467e-b35b-5b3210750fb0" containerID="c404790da86ee6c6dd0db3e96937d5c8adbcbf9beb1acd86b334dfa8677d7438" exitCode=0
Jul 10 18:55:53.572928 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.572788    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm" event={"ID":"54c9aabd-65b8-467e-b35b-5b3210750fb0","Type":"ContainerDied","Data":"c404790da86ee6c6dd0db3e96937d5c8adbcbf9beb1acd86b334dfa8677d7438"}
Jul 10 18:55:53.649626 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.649595    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:53.727556 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.727521    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c9aabd-65b8-467e-b35b-5b3210750fb0-tls-certs\") pod \"54c9aabd-65b8-467e-b35b-5b3210750fb0\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") "
Jul 10 18:55:53.727556 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.727561    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-dshm\") pod \"54c9aabd-65b8-467e-b35b-5b3210750fb0\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") "
Jul 10 18:55:53.727811 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.727608    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-kserve-provision-location\") pod \"54c9aabd-65b8-467e-b35b-5b3210750fb0\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") "
Jul 10 18:55:53.727811 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.727631    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-home\") pod \"54c9aabd-65b8-467e-b35b-5b3210750fb0\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") "
Jul 10 18:55:53.727811 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.727656    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-model-cache\") pod \"54c9aabd-65b8-467e-b35b-5b3210750fb0\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") "
Jul 10 18:55:53.727811 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.727690    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzxdv\" (UniqueName: \"kubernetes.io/projected/54c9aabd-65b8-467e-b35b-5b3210750fb0-kube-api-access-dzxdv\") pod \"54c9aabd-65b8-467e-b35b-5b3210750fb0\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") "
Jul 10 18:55:53.727811 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.727750    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-tmp-dir\") pod \"54c9aabd-65b8-467e-b35b-5b3210750fb0\" (UID: \"54c9aabd-65b8-467e-b35b-5b3210750fb0\") "
Jul 10 18:55:53.728104 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.728025    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-model-cache" (OuterVolumeSpecName: "model-cache") pod "54c9aabd-65b8-467e-b35b-5b3210750fb0" (UID: "54c9aabd-65b8-467e-b35b-5b3210750fb0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:53.728104 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.728034    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-home" (OuterVolumeSpecName: "home") pod "54c9aabd-65b8-467e-b35b-5b3210750fb0" (UID: "54c9aabd-65b8-467e-b35b-5b3210750fb0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:53.728258 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.728233    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54c9aabd-65b8-467e-b35b-5b3210750fb0" (UID: "54c9aabd-65b8-467e-b35b-5b3210750fb0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:53.729914 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.729884    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-dshm" (OuterVolumeSpecName: "dshm") pod "54c9aabd-65b8-467e-b35b-5b3210750fb0" (UID: "54c9aabd-65b8-467e-b35b-5b3210750fb0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:53.730338 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.730318    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54c9aabd-65b8-467e-b35b-5b3210750fb0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54c9aabd-65b8-467e-b35b-5b3210750fb0" (UID: "54c9aabd-65b8-467e-b35b-5b3210750fb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:55:53.730410 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.730360    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c9aabd-65b8-467e-b35b-5b3210750fb0-kube-api-access-dzxdv" (OuterVolumeSpecName: "kube-api-access-dzxdv") pod "54c9aabd-65b8-467e-b35b-5b3210750fb0" (UID: "54c9aabd-65b8-467e-b35b-5b3210750fb0"). InnerVolumeSpecName "kube-api-access-dzxdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:55:53.787519 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.787475    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "54c9aabd-65b8-467e-b35b-5b3210750fb0" (UID: "54c9aabd-65b8-467e-b35b-5b3210750fb0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:53.828486 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.828451    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-tmp-dir\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:53.828486 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.828479    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c9aabd-65b8-467e-b35b-5b3210750fb0-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:53.828486 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.828490    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-dshm\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:53.828713 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.828499    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-kserve-provision-location\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:53.828713 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.828507    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-home\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:53.828713 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.828515    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c9aabd-65b8-467e-b35b-5b3210750fb0-model-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:53.828713 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:53.828524    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzxdv\" (UniqueName: \"kubernetes.io/projected/54c9aabd-65b8-467e-b35b-5b3210750fb0-kube-api-access-dzxdv\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:54.578979 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:54.578948    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"
Jul 10 18:55:54.579462 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:54.578977    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm" event={"ID":"54c9aabd-65b8-467e-b35b-5b3210750fb0","Type":"ContainerDied","Data":"adb08dc8bbd1d75a0e2efe733a0ee5bdf085c147d040934e83a1878c10f4c771"}
Jul 10 18:55:54.579462 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:54.579050    2561 scope.go:117] "RemoveContainer" containerID="c404790da86ee6c6dd0db3e96937d5c8adbcbf9beb1acd86b334dfa8677d7438"
Jul 10 18:55:54.589711 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:54.589690    2561 scope.go:117] "RemoveContainer" containerID="4df42683ac717212b409400dc424500f48acfcd6bb912bd1e18793795c39b8d5"
Jul 10 18:55:54.613484 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:54.613437    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"]
Jul 10 18:55:54.616919 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:54.616881    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm"]
Jul 10 18:55:54.946534 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:54.946505    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:55.038448 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.038413    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d853843-1cf6-4292-8c25-321aaaea25d3-tls-certs\") pod \"9d853843-1cf6-4292-8c25-321aaaea25d3\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") "
Jul 10 18:55:55.038644 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.038476    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-tokenizer-tmp\") pod \"9d853843-1cf6-4292-8c25-321aaaea25d3\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") "
Jul 10 18:55:55.038644 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.038616    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl48c\" (UniqueName: \"kubernetes.io/projected/9d853843-1cf6-4292-8c25-321aaaea25d3-kube-api-access-tl48c\") pod \"9d853843-1cf6-4292-8c25-321aaaea25d3\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") "
Jul 10 18:55:55.038770 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.038673    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-tokenizer-uds\") pod \"9d853843-1cf6-4292-8c25-321aaaea25d3\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") "
Jul 10 18:55:55.038770 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.038705    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-kserve-provision-location\") pod \"9d853843-1cf6-4292-8c25-321aaaea25d3\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") "
Jul 10 18:55:55.038770 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.038729    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-tokenizer-cache\") pod \"9d853843-1cf6-4292-8c25-321aaaea25d3\" (UID: \"9d853843-1cf6-4292-8c25-321aaaea25d3\") "
Jul 10 18:55:55.038923 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.038836    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "9d853843-1cf6-4292-8c25-321aaaea25d3" (UID: "9d853843-1cf6-4292-8c25-321aaaea25d3"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:55.038992 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.038955    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "9d853843-1cf6-4292-8c25-321aaaea25d3" (UID: "9d853843-1cf6-4292-8c25-321aaaea25d3"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:55.039113 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.039063    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-tokenizer-tmp\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:55.039190 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.039118    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-tokenizer-uds\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:55.039324 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.039293    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "9d853843-1cf6-4292-8c25-321aaaea25d3" (UID: "9d853843-1cf6-4292-8c25-321aaaea25d3"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:55.039527 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.039505    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d853843-1cf6-4292-8c25-321aaaea25d3" (UID: "9d853843-1cf6-4292-8c25-321aaaea25d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:55.040656 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.040638    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d853843-1cf6-4292-8c25-321aaaea25d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d853843-1cf6-4292-8c25-321aaaea25d3" (UID: "9d853843-1cf6-4292-8c25-321aaaea25d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:55:55.040752 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.040734    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d853843-1cf6-4292-8c25-321aaaea25d3-kube-api-access-tl48c" (OuterVolumeSpecName: "kube-api-access-tl48c") pod "9d853843-1cf6-4292-8c25-321aaaea25d3" (UID: "9d853843-1cf6-4292-8c25-321aaaea25d3"). InnerVolumeSpecName "kube-api-access-tl48c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:55:55.072946 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.072913    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54c9aabd-65b8-467e-b35b-5b3210750fb0" path="/var/lib/kubelet/pods/54c9aabd-65b8-467e-b35b-5b3210750fb0/volumes"
Jul 10 18:55:55.140142 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.140027    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d853843-1cf6-4292-8c25-321aaaea25d3-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:55.140142 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.140062    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl48c\" (UniqueName: \"kubernetes.io/projected/9d853843-1cf6-4292-8c25-321aaaea25d3-kube-api-access-tl48c\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:55.140142 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.140103    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-kserve-provision-location\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:55.140142 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.140115    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d853843-1cf6-4292-8c25-321aaaea25d3-tokenizer-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:55.585977 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.585941    2561 generic.go:356] "Generic (PLEG): container finished" podID="9d853843-1cf6-4292-8c25-321aaaea25d3" containerID="e17bfe10bb499535ab130f0982f9564eaa6a9a79b7bb6f9436aa695c76380abc" exitCode=0
Jul 10 18:55:55.586408 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.586006    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"
Jul 10 18:55:55.586408 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.586028    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589" event={"ID":"9d853843-1cf6-4292-8c25-321aaaea25d3","Type":"ContainerDied","Data":"e17bfe10bb499535ab130f0982f9564eaa6a9a79b7bb6f9436aa695c76380abc"}
Jul 10 18:55:55.586408 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.586100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589" event={"ID":"9d853843-1cf6-4292-8c25-321aaaea25d3","Type":"ContainerDied","Data":"2ec8fa0cf33cf8baa1251def167a31efaf7d821ab7da72f27e5bf99de8b0cb37"}
Jul 10 18:55:55.586408 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.586127    2561 scope.go:117] "RemoveContainer" containerID="e17bfe10bb499535ab130f0982f9564eaa6a9a79b7bb6f9436aa695c76380abc"
Jul 10 18:55:55.594434 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.594416    2561 scope.go:117] "RemoveContainer" containerID="7d53e1fb283f565167a91177a3e2f280b3694e74efba3e1fdd1d05482331623c"
Jul 10 18:55:55.601997 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.601864    2561 scope.go:117] "RemoveContainer" containerID="77ebc48b068846ab77606c1ca191cceebc8580658f3cbe8d48d6c0de25659592"
Jul 10 18:55:55.609469 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.609442    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"]
Jul 10 18:55:55.610571 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.610551    2561 scope.go:117] "RemoveContainer" containerID="e17bfe10bb499535ab130f0982f9564eaa6a9a79b7bb6f9436aa695c76380abc"
Jul 10 18:55:55.610849 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:55:55.610828    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e17bfe10bb499535ab130f0982f9564eaa6a9a79b7bb6f9436aa695c76380abc\": container with ID starting with e17bfe10bb499535ab130f0982f9564eaa6a9a79b7bb6f9436aa695c76380abc not found: ID does not exist" containerID="e17bfe10bb499535ab130f0982f9564eaa6a9a79b7bb6f9436aa695c76380abc"
Jul 10 18:55:55.610910 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.610859    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e17bfe10bb499535ab130f0982f9564eaa6a9a79b7bb6f9436aa695c76380abc"} err="failed to get container status \"e17bfe10bb499535ab130f0982f9564eaa6a9a79b7bb6f9436aa695c76380abc\": rpc error: code = NotFound desc = could not find container \"e17bfe10bb499535ab130f0982f9564eaa6a9a79b7bb6f9436aa695c76380abc\": container with ID starting with e17bfe10bb499535ab130f0982f9564eaa6a9a79b7bb6f9436aa695c76380abc not found: ID does not exist"
Jul 10 18:55:55.610910 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.610879    2561 scope.go:117] "RemoveContainer" containerID="7d53e1fb283f565167a91177a3e2f280b3694e74efba3e1fdd1d05482331623c"
Jul 10 18:55:55.611181 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:55:55.611160    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d53e1fb283f565167a91177a3e2f280b3694e74efba3e1fdd1d05482331623c\": container with ID starting with 7d53e1fb283f565167a91177a3e2f280b3694e74efba3e1fdd1d05482331623c not found: ID does not exist" containerID="7d53e1fb283f565167a91177a3e2f280b3694e74efba3e1fdd1d05482331623c"
Jul 10 18:55:55.611257 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.611189    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d53e1fb283f565167a91177a3e2f280b3694e74efba3e1fdd1d05482331623c"} err="failed to get container status \"7d53e1fb283f565167a91177a3e2f280b3694e74efba3e1fdd1d05482331623c\": rpc error: code = NotFound desc = could not find container \"7d53e1fb283f565167a91177a3e2f280b3694e74efba3e1fdd1d05482331623c\": container with ID starting with 7d53e1fb283f565167a91177a3e2f280b3694e74efba3e1fdd1d05482331623c not found: ID does not exist"
Jul 10 18:55:55.611257 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.611211    2561 scope.go:117] "RemoveContainer" containerID="77ebc48b068846ab77606c1ca191cceebc8580658f3cbe8d48d6c0de25659592"
Jul 10 18:55:55.614016 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.611607    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589"]
Jul 10 18:55:55.614172 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:55:55.614148    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ebc48b068846ab77606c1ca191cceebc8580658f3cbe8d48d6c0de25659592\": container with ID starting with 77ebc48b068846ab77606c1ca191cceebc8580658f3cbe8d48d6c0de25659592 not found: ID does not exist" containerID="77ebc48b068846ab77606c1ca191cceebc8580658f3cbe8d48d6c0de25659592"
Jul 10 18:55:55.614231 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:55.614183    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ebc48b068846ab77606c1ca191cceebc8580658f3cbe8d48d6c0de25659592"} err="failed to get container status \"77ebc48b068846ab77606c1ca191cceebc8580658f3cbe8d48d6c0de25659592\": rpc error: code = NotFound desc = could not find container \"77ebc48b068846ab77606c1ca191cceebc8580658f3cbe8d48d6c0de25659592\": container with ID starting with 77ebc48b068846ab77606c1ca191cceebc8580658f3cbe8d48d6c0de25659592 not found: ID does not exist"
Jul 10 18:55:56.648771 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.648721    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:55:56.906336 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.906297    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"]
Jul 10 18:55:56.906678 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.906666    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54c9aabd-65b8-467e-b35b-5b3210750fb0" containerName="main"
Jul 10 18:55:56.906734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.906680    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="54c9aabd-65b8-467e-b35b-5b3210750fb0" containerName="main"
Jul 10 18:55:56.906734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.906689    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54c9aabd-65b8-467e-b35b-5b3210750fb0" containerName="storage-initializer"
Jul 10 18:55:56.906734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.906695    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="54c9aabd-65b8-467e-b35b-5b3210750fb0" containerName="storage-initializer"
Jul 10 18:55:56.906734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.906708    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d853843-1cf6-4292-8c25-321aaaea25d3" containerName="tokenizer"
Jul 10 18:55:56.906734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.906714    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d853843-1cf6-4292-8c25-321aaaea25d3" containerName="tokenizer"
Jul 10 18:55:56.906734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.906729    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d853843-1cf6-4292-8c25-321aaaea25d3" containerName="main"
Jul 10 18:55:56.906734 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.906734    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d853843-1cf6-4292-8c25-321aaaea25d3" containerName="main"
Jul 10 18:55:56.906976 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.906742    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d853843-1cf6-4292-8c25-321aaaea25d3" containerName="storage-initializer"
Jul 10 18:55:56.906976 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.906747    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d853843-1cf6-4292-8c25-321aaaea25d3" containerName="storage-initializer"
Jul 10 18:55:56.906976 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.906803    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d853843-1cf6-4292-8c25-321aaaea25d3" containerName="main"
Jul 10 18:55:56.906976 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.906814    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d853843-1cf6-4292-8c25-321aaaea25d3" containerName="tokenizer"
Jul 10 18:55:56.906976 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.906821    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="54c9aabd-65b8-467e-b35b-5b3210750fb0" containerName="main"
Jul 10 18:55:56.911554 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.911530    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:56.914291 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.914267    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 18:55:56.914445 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.914372    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 18:55:56.915196 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.915165    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 10 18:55:56.915196 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.915182    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jjdpb\""
Jul 10 18:55:56.915196 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.915182    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-w2jz5\""
Jul 10 18:55:56.921253 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:56.921226    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"]
Jul 10 18:55:57.006309 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.006243    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:55:57.056347 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.056307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.056522 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.056358    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdwlf\" (UniqueName: \"kubernetes.io/projected/328d22d6-b179-4b73-91e3-20b8b077adad-kube-api-access-vdwlf\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.056522 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.056389    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.056522 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.056466    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328d22d6-b179-4b73-91e3-20b8b077adad-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.056522 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.056497    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.056663 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.056524    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.073857 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.073828    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d853843-1cf6-4292-8c25-321aaaea25d3" path="/var/lib/kubelet/pods/9d853843-1cf6-4292-8c25-321aaaea25d3/volumes"
Jul 10 18:55:57.157604 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.157520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328d22d6-b179-4b73-91e3-20b8b077adad-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.157604 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.157559    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.157604 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.157588    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.157982 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.157640    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.157982 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.157678    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdwlf\" (UniqueName: \"kubernetes.io/projected/328d22d6-b179-4b73-91e3-20b8b077adad-kube-api-access-vdwlf\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.157982 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.157710    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.158147 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.158002    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.158147 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.158134    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.158223 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.158152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.158223 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.158193    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.160337 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.160317    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328d22d6-b179-4b73-91e3-20b8b077adad-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.167913 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.167882    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdwlf\" (UniqueName: \"kubernetes.io/projected/328d22d6-b179-4b73-91e3-20b8b077adad-kube-api-access-vdwlf\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.221699 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.221657    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:57.365551 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.365514    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"]
Jul 10 18:55:57.368241 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:55:57.368211    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328d22d6_b179_4b73_91e3_20b8b077adad.slice/crio-11f4dec97347b6d2ef0dd54813fa66d6089caefa7e67dff677f776ab50dfd5bc WatchSource:0}: Error finding container 11f4dec97347b6d2ef0dd54813fa66d6089caefa7e67dff677f776ab50dfd5bc: Status 404 returned error can't find the container with id 11f4dec97347b6d2ef0dd54813fa66d6089caefa7e67dff677f776ab50dfd5bc
Jul 10 18:55:57.598774 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.598738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg" event={"ID":"328d22d6-b179-4b73-91e3-20b8b077adad","Type":"ContainerStarted","Data":"3332cc0593abe0e37bc049cc159900cf5a754f127de2181d1bdfac71f32081a0"}
Jul 10 18:55:57.598983 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:57.598796    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg" event={"ID":"328d22d6-b179-4b73-91e3-20b8b077adad","Type":"ContainerStarted","Data":"11f4dec97347b6d2ef0dd54813fa66d6089caefa7e67dff677f776ab50dfd5bc"}
Jul 10 18:55:58.605060 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:58.605022    2561 generic.go:356] "Generic (PLEG): container finished" podID="328d22d6-b179-4b73-91e3-20b8b077adad" containerID="3332cc0593abe0e37bc049cc159900cf5a754f127de2181d1bdfac71f32081a0" exitCode=0
Jul 10 18:55:58.605574 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:58.605116    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg" event={"ID":"328d22d6-b179-4b73-91e3-20b8b077adad","Type":"ContainerDied","Data":"3332cc0593abe0e37bc049cc159900cf5a754f127de2181d1bdfac71f32081a0"}
Jul 10 18:55:59.612465 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:59.612375    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg" event={"ID":"328d22d6-b179-4b73-91e3-20b8b077adad","Type":"ContainerStarted","Data":"4e0ebde81e786788d1ec940791f9f0209048fe8598c2cb1d9eaf7aed02fcfea6"}
Jul 10 18:55:59.612465 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:59.612430    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg" event={"ID":"328d22d6-b179-4b73-91e3-20b8b077adad","Type":"ContainerStarted","Data":"2876279db0c8cabfc0ca5b41ebc1013f5951c6558010cd6c865a58fdf3cc9d45"}
Jul 10 18:55:59.612465 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:59.612467    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:55:59.636902 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:55:59.636843    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg" podStartSLOduration=2.9591713779999997 podStartE2EDuration="3.636823811s" totalImagesPullingTime="0s" totalInitContainerRuntime="677.652433ms" isStatefulPod=false podCreationTimestamp="2026-07-10 18:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:55:59.636350191 +0000 UTC m=+983.110443075" watchObservedRunningTime="2026-07-10 18:55:59.636823811 +0000 UTC m=+983.110916727"
Jul 10 18:56:07.222446 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:56:07.222407    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:56:07.222880 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:56:07.222488    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:56:07.225315 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:56:07.225290    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:56:07.345878 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:56:07.345837    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:56:07.657189 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:56:07.657161    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:56:28.662933 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:56:28.662896    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:57:49.222197 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:57:49.222114    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:00.675597 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:00.675559    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:00.776767 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:00.776728    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:04.483163 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:04.483126    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:04.801270 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:04.801154    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:12.846155 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:12.846120    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:15.705594 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:15.705555    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:15.782486 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:15.782443    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"]
Jul 10 18:58:15.782754 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:15.782723    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg" podUID="328d22d6-b179-4b73-91e3-20b8b077adad" containerName="main" containerID="cri-o://2876279db0c8cabfc0ca5b41ebc1013f5951c6558010cd6c865a58fdf3cc9d45" gracePeriod=30
Jul 10 18:58:15.782848 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:15.782811    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg" podUID="328d22d6-b179-4b73-91e3-20b8b077adad" containerName="tokenizer" containerID="cri-o://4e0ebde81e786788d1ec940791f9f0209048fe8598c2cb1d9eaf7aed02fcfea6" gracePeriod=30
Jul 10 18:58:15.814408 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:15.814363    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:15.913331 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:15.913293    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:16.259627 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:16.259585    2561 generic.go:356] "Generic (PLEG): container finished" podID="328d22d6-b179-4b73-91e3-20b8b077adad" containerID="2876279db0c8cabfc0ca5b41ebc1013f5951c6558010cd6c865a58fdf3cc9d45" exitCode=0
Jul 10 18:58:16.259627 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:16.259610    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg" event={"ID":"328d22d6-b179-4b73-91e3-20b8b077adad","Type":"ContainerDied","Data":"2876279db0c8cabfc0ca5b41ebc1013f5951c6558010cd6c865a58fdf3cc9d45"}
Jul 10 18:58:17.142883 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.142855    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:58:17.266985 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.266884    2561 generic.go:356] "Generic (PLEG): container finished" podID="328d22d6-b179-4b73-91e3-20b8b077adad" containerID="4e0ebde81e786788d1ec940791f9f0209048fe8598c2cb1d9eaf7aed02fcfea6" exitCode=0
Jul 10 18:58:17.266985 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.266951    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"
Jul 10 18:58:17.266985 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.266960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg" event={"ID":"328d22d6-b179-4b73-91e3-20b8b077adad","Type":"ContainerDied","Data":"4e0ebde81e786788d1ec940791f9f0209048fe8598c2cb1d9eaf7aed02fcfea6"}
Jul 10 18:58:17.267283 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.267037    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg" event={"ID":"328d22d6-b179-4b73-91e3-20b8b077adad","Type":"ContainerDied","Data":"11f4dec97347b6d2ef0dd54813fa66d6089caefa7e67dff677f776ab50dfd5bc"}
Jul 10 18:58:17.267283 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.267071    2561 scope.go:117] "RemoveContainer" containerID="4e0ebde81e786788d1ec940791f9f0209048fe8598c2cb1d9eaf7aed02fcfea6"
Jul 10 18:58:17.277370 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.277346    2561 scope.go:117] "RemoveContainer" containerID="2876279db0c8cabfc0ca5b41ebc1013f5951c6558010cd6c865a58fdf3cc9d45"
Jul 10 18:58:17.279700 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.279677    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-tokenizer-tmp\") pod \"328d22d6-b179-4b73-91e3-20b8b077adad\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") "
Jul 10 18:58:17.279811 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.279777    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdwlf\" (UniqueName: \"kubernetes.io/projected/328d22d6-b179-4b73-91e3-20b8b077adad-kube-api-access-vdwlf\") pod \"328d22d6-b179-4b73-91e3-20b8b077adad\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") "
Jul 10 18:58:17.279811 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.279803    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328d22d6-b179-4b73-91e3-20b8b077adad-tls-certs\") pod \"328d22d6-b179-4b73-91e3-20b8b077adad\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") "
Jul 10 18:58:17.280628 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.279835    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-kserve-provision-location\") pod \"328d22d6-b179-4b73-91e3-20b8b077adad\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") "
Jul 10 18:58:17.280628 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.279891    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-tokenizer-cache\") pod \"328d22d6-b179-4b73-91e3-20b8b077adad\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") "
Jul 10 18:58:17.280628 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.279977    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-tokenizer-uds\") pod \"328d22d6-b179-4b73-91e3-20b8b077adad\" (UID: \"328d22d6-b179-4b73-91e3-20b8b077adad\") "
Jul 10 18:58:17.280628 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.280049    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "328d22d6-b179-4b73-91e3-20b8b077adad" (UID: "328d22d6-b179-4b73-91e3-20b8b077adad"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:58:17.280628 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.280350    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-tokenizer-tmp\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:58:17.280628 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.280444    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "328d22d6-b179-4b73-91e3-20b8b077adad" (UID: "328d22d6-b179-4b73-91e3-20b8b077adad"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:58:17.280628 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.280525    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "328d22d6-b179-4b73-91e3-20b8b077adad" (UID: "328d22d6-b179-4b73-91e3-20b8b077adad"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:58:17.280962 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.280921    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "328d22d6-b179-4b73-91e3-20b8b077adad" (UID: "328d22d6-b179-4b73-91e3-20b8b077adad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:58:17.282515 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.282491    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328d22d6-b179-4b73-91e3-20b8b077adad-kube-api-access-vdwlf" (OuterVolumeSpecName: "kube-api-access-vdwlf") pod "328d22d6-b179-4b73-91e3-20b8b077adad" (UID: "328d22d6-b179-4b73-91e3-20b8b077adad"). InnerVolumeSpecName "kube-api-access-vdwlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:58:17.282807 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.282785    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/328d22d6-b179-4b73-91e3-20b8b077adad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "328d22d6-b179-4b73-91e3-20b8b077adad" (UID: "328d22d6-b179-4b73-91e3-20b8b077adad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:58:17.289532 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.288989    2561 scope.go:117] "RemoveContainer" containerID="3332cc0593abe0e37bc049cc159900cf5a754f127de2181d1bdfac71f32081a0"
Jul 10 18:58:17.304185 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.304164    2561 scope.go:117] "RemoveContainer" containerID="4e0ebde81e786788d1ec940791f9f0209048fe8598c2cb1d9eaf7aed02fcfea6"
Jul 10 18:58:17.304534 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:58:17.304502    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e0ebde81e786788d1ec940791f9f0209048fe8598c2cb1d9eaf7aed02fcfea6\": container with ID starting with 4e0ebde81e786788d1ec940791f9f0209048fe8598c2cb1d9eaf7aed02fcfea6 not found: ID does not exist" containerID="4e0ebde81e786788d1ec940791f9f0209048fe8598c2cb1d9eaf7aed02fcfea6"
Jul 10 18:58:17.304594 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.304544    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e0ebde81e786788d1ec940791f9f0209048fe8598c2cb1d9eaf7aed02fcfea6"} err="failed to get container status \"4e0ebde81e786788d1ec940791f9f0209048fe8598c2cb1d9eaf7aed02fcfea6\": rpc error: code = NotFound desc = could not find container \"4e0ebde81e786788d1ec940791f9f0209048fe8598c2cb1d9eaf7aed02fcfea6\": container with ID starting with 4e0ebde81e786788d1ec940791f9f0209048fe8598c2cb1d9eaf7aed02fcfea6 not found: ID does not exist"
Jul 10 18:58:17.304594 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.304565    2561 scope.go:117] "RemoveContainer" containerID="2876279db0c8cabfc0ca5b41ebc1013f5951c6558010cd6c865a58fdf3cc9d45"
Jul 10 18:58:17.304878 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:58:17.304855    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2876279db0c8cabfc0ca5b41ebc1013f5951c6558010cd6c865a58fdf3cc9d45\": container with ID starting with 2876279db0c8cabfc0ca5b41ebc1013f5951c6558010cd6c865a58fdf3cc9d45 not found: ID does not exist" containerID="2876279db0c8cabfc0ca5b41ebc1013f5951c6558010cd6c865a58fdf3cc9d45"
Jul 10 18:58:17.304920 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.304888    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2876279db0c8cabfc0ca5b41ebc1013f5951c6558010cd6c865a58fdf3cc9d45"} err="failed to get container status \"2876279db0c8cabfc0ca5b41ebc1013f5951c6558010cd6c865a58fdf3cc9d45\": rpc error: code = NotFound desc = could not find container \"2876279db0c8cabfc0ca5b41ebc1013f5951c6558010cd6c865a58fdf3cc9d45\": container with ID starting with 2876279db0c8cabfc0ca5b41ebc1013f5951c6558010cd6c865a58fdf3cc9d45 not found: ID does not exist"
Jul 10 18:58:17.304920 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.304910    2561 scope.go:117] "RemoveContainer" containerID="3332cc0593abe0e37bc049cc159900cf5a754f127de2181d1bdfac71f32081a0"
Jul 10 18:58:17.305265 ip-10-0-135-91 kubenswrapper[2561]: E0710 18:58:17.305239    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3332cc0593abe0e37bc049cc159900cf5a754f127de2181d1bdfac71f32081a0\": container with ID starting with 3332cc0593abe0e37bc049cc159900cf5a754f127de2181d1bdfac71f32081a0 not found: ID does not exist" containerID="3332cc0593abe0e37bc049cc159900cf5a754f127de2181d1bdfac71f32081a0"
Jul 10 18:58:17.305371 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.305270    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3332cc0593abe0e37bc049cc159900cf5a754f127de2181d1bdfac71f32081a0"} err="failed to get container status \"3332cc0593abe0e37bc049cc159900cf5a754f127de2181d1bdfac71f32081a0\": rpc error: code = NotFound desc = could not find container \"3332cc0593abe0e37bc049cc159900cf5a754f127de2181d1bdfac71f32081a0\": container with ID starting with 3332cc0593abe0e37bc049cc159900cf5a754f127de2181d1bdfac71f32081a0 not found: ID does not exist"
Jul 10 18:58:17.381593 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.381551    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-tokenizer-uds\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:58:17.381593 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.381584    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdwlf\" (UniqueName: \"kubernetes.io/projected/328d22d6-b179-4b73-91e3-20b8b077adad-kube-api-access-vdwlf\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:58:17.381593 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.381597    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328d22d6-b179-4b73-91e3-20b8b077adad-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:58:17.381830 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.381612    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-kserve-provision-location\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:58:17.381830 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.381622    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/328d22d6-b179-4b73-91e3-20b8b077adad-tokenizer-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 18:58:17.591071 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.591037    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"]
Jul 10 18:58:17.595037 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:17.595013    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg"]
Jul 10 18:58:19.073666 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:19.073631    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="328d22d6-b179-4b73-91e3-20b8b077adad" path="/var/lib/kubelet/pods/328d22d6-b179-4b73-91e3-20b8b077adad/volumes"
Jul 10 18:58:36.368875 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.368832    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:36.535907 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.535876    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"]
Jul 10 18:58:36.536292 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.536277    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328d22d6-b179-4b73-91e3-20b8b077adad" containerName="tokenizer"
Jul 10 18:58:36.536292 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.536292    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="328d22d6-b179-4b73-91e3-20b8b077adad" containerName="tokenizer"
Jul 10 18:58:36.536422 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.536321    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328d22d6-b179-4b73-91e3-20b8b077adad" containerName="storage-initializer"
Jul 10 18:58:36.536422 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.536328    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="328d22d6-b179-4b73-91e3-20b8b077adad" containerName="storage-initializer"
Jul 10 18:58:36.536422 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.536333    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328d22d6-b179-4b73-91e3-20b8b077adad" containerName="main"
Jul 10 18:58:36.536422 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.536339    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="328d22d6-b179-4b73-91e3-20b8b077adad" containerName="main"
Jul 10 18:58:36.536422 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.536403    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="328d22d6-b179-4b73-91e3-20b8b077adad" containerName="main"
Jul 10 18:58:36.536422 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.536415    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="328d22d6-b179-4b73-91e3-20b8b077adad" containerName="tokenizer"
Jul 10 18:58:36.539746 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.539684    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.542314 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.542289    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 18:58:36.542449 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.542363    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 18:58:36.542525 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.542371    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-r8spk\""
Jul 10 18:58:36.542525 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.542391    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jjdpb\""
Jul 10 18:58:36.543183 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.543165    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 10 18:58:36.555595 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.555570    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"]
Jul 10 18:58:36.579341 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.579305    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:36.659568 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.659540    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7j6p\" (UniqueName: \"kubernetes.io/projected/8744787d-10c1-476a-8494-856ac64893a9-kube-api-access-d7j6p\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.659732 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.659584    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.659732 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.659613    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.659732 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.659667    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.659846 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.659745    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.659846 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.659803    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8744787d-10c1-476a-8494-856ac64893a9-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.760814 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.760769    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.760973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.760826    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8744787d-10c1-476a-8494-856ac64893a9-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.760973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.760874    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7j6p\" (UniqueName: \"kubernetes.io/projected/8744787d-10c1-476a-8494-856ac64893a9-kube-api-access-d7j6p\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.760973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.760914    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.760973 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.760954    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.761230 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.760980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.761322 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.761297    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.761389 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.761317    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.761389 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.761346    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.761485 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.761406    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.763348 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.763331    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8744787d-10c1-476a-8494-856ac64893a9-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.772661 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.772628    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7j6p\" (UniqueName: \"kubernetes.io/projected/8744787d-10c1-476a-8494-856ac64893a9-kube-api-access-d7j6p\") pod \"stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.850396 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.850354    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:36.992232 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:36.992206    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"]
Jul 10 18:58:36.994507 ip-10-0-135-91 kubenswrapper[2561]: W0710 18:58:36.994475    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8744787d_10c1_476a_8494_856ac64893a9.slice/crio-0fd99735b738646f219f15c71ff269979013233429d534efe4c6c8787854d9d8 WatchSource:0}: Error finding container 0fd99735b738646f219f15c71ff269979013233429d534efe4c6c8787854d9d8: Status 404 returned error can't find the container with id 0fd99735b738646f219f15c71ff269979013233429d534efe4c6c8787854d9d8
Jul 10 18:58:37.362318 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:37.362219    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b" event={"ID":"8744787d-10c1-476a-8494-856ac64893a9","Type":"ContainerStarted","Data":"7c22487a5f60f9f374595a29349ffe5253a49b643b02e64b755241a10b75d179"}
Jul 10 18:58:37.362318 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:37.362308    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b" event={"ID":"8744787d-10c1-476a-8494-856ac64893a9","Type":"ContainerStarted","Data":"0fd99735b738646f219f15c71ff269979013233429d534efe4c6c8787854d9d8"}
Jul 10 18:58:38.368794 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:38.368760    2561 generic.go:356] "Generic (PLEG): container finished" podID="8744787d-10c1-476a-8494-856ac64893a9" containerID="7c22487a5f60f9f374595a29349ffe5253a49b643b02e64b755241a10b75d179" exitCode=0
Jul 10 18:58:38.369203 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:38.368826    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b" event={"ID":"8744787d-10c1-476a-8494-856ac64893a9","Type":"ContainerDied","Data":"7c22487a5f60f9f374595a29349ffe5253a49b643b02e64b755241a10b75d179"}
Jul 10 18:58:39.040287 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:39.040224    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:39.375337 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:39.375247    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b" event={"ID":"8744787d-10c1-476a-8494-856ac64893a9","Type":"ContainerStarted","Data":"8c3132b4746106219d36a3c2eca8727dffe6489e2cc622bc0beb3898a14d322a"}
Jul 10 18:58:39.375337 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:39.375313    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b" event={"ID":"8744787d-10c1-476a-8494-856ac64893a9","Type":"ContainerStarted","Data":"83d43568034800b1b68bc6a3ab5240ed624642f1575b78ffada1ee00ddd96efd"}
Jul 10 18:58:39.375764 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:39.375376    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:39.407493 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:39.407439    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b" podStartSLOduration=2.7221515370000002 podStartE2EDuration="3.407421217s" totalImagesPullingTime="0s" totalInitContainerRuntime="685.26968ms" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:39.406808295 +0000 UTC m=+1142.880901200" watchObservedRunningTime="2026-07-10 18:58:39.407421217 +0000 UTC m=+1142.881514241"
Jul 10 18:58:46.770648 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:46.770611    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 18:58:46.851322 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:46.851281    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:46.851530 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:46.851359    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:46.854145 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:46.854119    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:58:47.424008 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:58:47.423977    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:59:08.429699 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:59:08.429627    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 18:59:37.062223 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:59:37.062193    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 18:59:37.065115 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:59:37.065070    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 18:59:37.070384 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:59:37.070362    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 18:59:37.073463 ip-10-0-135-91 kubenswrapper[2561]: I0710 18:59:37.073439    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 19:00:13.931329 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:13.931290    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:14.027288 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:14.027252    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:17.259732 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.259685    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:17.513559 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.513453    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"]
Jul 10 19:00:17.517699 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.517674    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.520487 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.520461    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-l2dkd\""
Jul 10 19:00:17.520626 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.520588    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 10 19:00:17.531623 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.531587    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"]
Jul 10 19:00:17.547227 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.547193    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:17.649567 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.649532    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.649746 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.649589    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.649746 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.649629    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.649746 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.649660    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.649746 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.649691    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.649746 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.649745    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bptc9\" (UniqueName: \"kubernetes.io/projected/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-kube-api-access-bptc9\") pod \"custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.751003 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.750959    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.751220 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.751026    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.751220 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.751049    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.751220 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.751103    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.751220 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.751185    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.751455 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.751260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bptc9\" (UniqueName: \"kubernetes.io/projected/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-kube-api-access-bptc9\") pod \"custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.751515 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.751453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.751727 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.751690    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.751864 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.751742    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.751864 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.751853    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.753658 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.753641    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.763193 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.763162    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bptc9\" (UniqueName: \"kubernetes.io/projected/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-kube-api-access-bptc9\") pod \"custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.829147 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.829053    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:17.975627 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:17.975602    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"]
Jul 10 19:00:17.977645 ip-10-0-135-91 kubenswrapper[2561]: W0710 19:00:17.977612    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd1ba36_ae63_4416_9a5c_0d2715af5b7d.slice/crio-0398ac3d70942cccd317e31cc0d0011475db9a8bee2c8ed40e1c165e95f78e75 WatchSource:0}: Error finding container 0398ac3d70942cccd317e31cc0d0011475db9a8bee2c8ed40e1c165e95f78e75: Status 404 returned error can't find the container with id 0398ac3d70942cccd317e31cc0d0011475db9a8bee2c8ed40e1c165e95f78e75
Jul 10 19:00:18.878764 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:18.878722    2561 generic.go:356] "Generic (PLEG): container finished" podID="5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" containerID="afa70468b99adcddc7c8ebbf14e7d459ee7767bf41f0f83583e7d46379bd9068" exitCode=0
Jul 10 19:00:18.879174 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:18.878771    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m" event={"ID":"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d","Type":"ContainerDied","Data":"afa70468b99adcddc7c8ebbf14e7d459ee7767bf41f0f83583e7d46379bd9068"}
Jul 10 19:00:18.879174 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:18.878855    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m" event={"ID":"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d","Type":"ContainerStarted","Data":"0398ac3d70942cccd317e31cc0d0011475db9a8bee2c8ed40e1c165e95f78e75"}
Jul 10 19:00:19.886614 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:19.886567    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m" event={"ID":"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d","Type":"ContainerStarted","Data":"40c4af77b06b4c108796e387e5404e2894104ff96c853302e88b673110543063"}
Jul 10 19:00:19.887010 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:19.886631    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m" event={"ID":"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d","Type":"ContainerStarted","Data":"89d17495ec092252ca0eee6f701e7c2c66e33756440cb0c40afd23f6367c2bcf"}
Jul 10 19:00:19.887010 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:19.886660    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:19.912877 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:19.912821    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m" podStartSLOduration=2.231064823 podStartE2EDuration="2.912805s" totalImagesPullingTime="0s" totalInitContainerRuntime="681.740177ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:19.910825751 +0000 UTC m=+1243.384918635" watchObservedRunningTime="2026-07-10 19:00:19.912805 +0000 UTC m=+1243.386897906"
Jul 10 19:00:25.768690 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:25.768646    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:27.829943 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:27.829901    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:27.830413 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:27.829971    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:27.832725 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:27.832701    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:27.925821 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:27.925795    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:00:37.903036 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:37.902949    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:37.926714 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:37.926677    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"]
Jul 10 19:00:37.926985 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:37.926959    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b" podUID="8744787d-10c1-476a-8494-856ac64893a9" containerName="main" containerID="cri-o://83d43568034800b1b68bc6a3ab5240ed624642f1575b78ffada1ee00ddd96efd" gracePeriod=30
Jul 10 19:00:37.927111 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:37.926995    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b" podUID="8744787d-10c1-476a-8494-856ac64893a9" containerName="tokenizer" containerID="cri-o://8c3132b4746106219d36a3c2eca8727dffe6489e2cc622bc0beb3898a14d322a" gracePeriod=30
Jul 10 19:00:38.001853 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:38.001808    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:38.428859 ip-10-0-135-91 kubenswrapper[2561]: W0710 19:00:38.428825    2561 logging.go:55] [core] [Channel #204 SubChannel #205]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.39:9003", ServerName: "10.132.0.39:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.39:9003: connect: connection refused"
Jul 10 19:00:38.977052 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:38.977016    2561 generic.go:356] "Generic (PLEG): container finished" podID="8744787d-10c1-476a-8494-856ac64893a9" containerID="83d43568034800b1b68bc6a3ab5240ed624642f1575b78ffada1ee00ddd96efd" exitCode=0
Jul 10 19:00:38.977529 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:38.977101    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b" event={"ID":"8744787d-10c1-476a-8494-856ac64893a9","Type":"ContainerDied","Data":"83d43568034800b1b68bc6a3ab5240ed624642f1575b78ffada1ee00ddd96efd"}
Jul 10 19:00:39.199216 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.199191    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 19:00:39.237949 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.237841    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-tokenizer-uds\") pod \"8744787d-10c1-476a-8494-856ac64893a9\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") "
Jul 10 19:00:39.237949 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.237916    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8744787d-10c1-476a-8494-856ac64893a9-tls-certs\") pod \"8744787d-10c1-476a-8494-856ac64893a9\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") "
Jul 10 19:00:39.238230 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.237950    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7j6p\" (UniqueName: \"kubernetes.io/projected/8744787d-10c1-476a-8494-856ac64893a9-kube-api-access-d7j6p\") pod \"8744787d-10c1-476a-8494-856ac64893a9\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") "
Jul 10 19:00:39.238230 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.237986    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-tokenizer-tmp\") pod \"8744787d-10c1-476a-8494-856ac64893a9\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") "
Jul 10 19:00:39.238230 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.238024    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-kserve-provision-location\") pod \"8744787d-10c1-476a-8494-856ac64893a9\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") "
Jul 10 19:00:39.238230 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.238119    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-tokenizer-cache\") pod \"8744787d-10c1-476a-8494-856ac64893a9\" (UID: \"8744787d-10c1-476a-8494-856ac64893a9\") "
Jul 10 19:00:39.238530 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.238496    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "8744787d-10c1-476a-8494-856ac64893a9" (UID: "8744787d-10c1-476a-8494-856ac64893a9"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:39.238819 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.238781    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "8744787d-10c1-476a-8494-856ac64893a9" (UID: "8744787d-10c1-476a-8494-856ac64893a9"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:39.238819 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.238796    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8744787d-10c1-476a-8494-856ac64893a9" (UID: "8744787d-10c1-476a-8494-856ac64893a9"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:39.239250 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.239220    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8744787d-10c1-476a-8494-856ac64893a9" (UID: "8744787d-10c1-476a-8494-856ac64893a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:39.241023 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.240995    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8744787d-10c1-476a-8494-856ac64893a9-kube-api-access-d7j6p" (OuterVolumeSpecName: "kube-api-access-d7j6p") pod "8744787d-10c1-476a-8494-856ac64893a9" (UID: "8744787d-10c1-476a-8494-856ac64893a9"). InnerVolumeSpecName "kube-api-access-d7j6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:00:39.241148 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.241071    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8744787d-10c1-476a-8494-856ac64893a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8744787d-10c1-476a-8494-856ac64893a9" (UID: "8744787d-10c1-476a-8494-856ac64893a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:39.339525 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.339490    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-tokenizer-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:39.339525 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.339519    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-tokenizer-uds\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:39.339525 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.339528    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8744787d-10c1-476a-8494-856ac64893a9-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:39.339766 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.339536    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7j6p\" (UniqueName: \"kubernetes.io/projected/8744787d-10c1-476a-8494-856ac64893a9-kube-api-access-d7j6p\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:39.339766 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.339545    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-tokenizer-tmp\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:39.339766 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.339553    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8744787d-10c1-476a-8494-856ac64893a9-kserve-provision-location\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:39.428860 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.428815    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b" podUID="8744787d-10c1-476a-8494-856ac64893a9" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.39:9003\" within 1s: context deadline exceeded"
Jul 10 19:00:39.983300 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.983263    2561 generic.go:356] "Generic (PLEG): container finished" podID="8744787d-10c1-476a-8494-856ac64893a9" containerID="8c3132b4746106219d36a3c2eca8727dffe6489e2cc622bc0beb3898a14d322a" exitCode=0
Jul 10 19:00:39.983695 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.983322    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"
Jul 10 19:00:39.983695 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.983340    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b" event={"ID":"8744787d-10c1-476a-8494-856ac64893a9","Type":"ContainerDied","Data":"8c3132b4746106219d36a3c2eca8727dffe6489e2cc622bc0beb3898a14d322a"}
Jul 10 19:00:39.983695 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.983396    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b" event={"ID":"8744787d-10c1-476a-8494-856ac64893a9","Type":"ContainerDied","Data":"0fd99735b738646f219f15c71ff269979013233429d534efe4c6c8787854d9d8"}
Jul 10 19:00:39.983695 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.983422    2561 scope.go:117] "RemoveContainer" containerID="8c3132b4746106219d36a3c2eca8727dffe6489e2cc622bc0beb3898a14d322a"
Jul 10 19:00:39.992660 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:39.992644    2561 scope.go:117] "RemoveContainer" containerID="83d43568034800b1b68bc6a3ab5240ed624642f1575b78ffada1ee00ddd96efd"
Jul 10 19:00:40.000521 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:40.000501    2561 scope.go:117] "RemoveContainer" containerID="7c22487a5f60f9f374595a29349ffe5253a49b643b02e64b755241a10b75d179"
Jul 10 19:00:40.008363 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:40.008345    2561 scope.go:117] "RemoveContainer" containerID="8c3132b4746106219d36a3c2eca8727dffe6489e2cc622bc0beb3898a14d322a"
Jul 10 19:00:40.008688 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:00:40.008668    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c3132b4746106219d36a3c2eca8727dffe6489e2cc622bc0beb3898a14d322a\": container with ID starting with 8c3132b4746106219d36a3c2eca8727dffe6489e2cc622bc0beb3898a14d322a not found: ID does not exist" containerID="8c3132b4746106219d36a3c2eca8727dffe6489e2cc622bc0beb3898a14d322a"
Jul 10 19:00:40.008779 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:40.008700    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3132b4746106219d36a3c2eca8727dffe6489e2cc622bc0beb3898a14d322a"} err="failed to get container status \"8c3132b4746106219d36a3c2eca8727dffe6489e2cc622bc0beb3898a14d322a\": rpc error: code = NotFound desc = could not find container \"8c3132b4746106219d36a3c2eca8727dffe6489e2cc622bc0beb3898a14d322a\": container with ID starting with 8c3132b4746106219d36a3c2eca8727dffe6489e2cc622bc0beb3898a14d322a not found: ID does not exist"
Jul 10 19:00:40.008779 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:40.008725    2561 scope.go:117] "RemoveContainer" containerID="83d43568034800b1b68bc6a3ab5240ed624642f1575b78ffada1ee00ddd96efd"
Jul 10 19:00:40.008952 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:00:40.008932    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d43568034800b1b68bc6a3ab5240ed624642f1575b78ffada1ee00ddd96efd\": container with ID starting with 83d43568034800b1b68bc6a3ab5240ed624642f1575b78ffada1ee00ddd96efd not found: ID does not exist" containerID="83d43568034800b1b68bc6a3ab5240ed624642f1575b78ffada1ee00ddd96efd"
Jul 10 19:00:40.008996 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:40.008958    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d43568034800b1b68bc6a3ab5240ed624642f1575b78ffada1ee00ddd96efd"} err="failed to get container status \"83d43568034800b1b68bc6a3ab5240ed624642f1575b78ffada1ee00ddd96efd\": rpc error: code = NotFound desc = could not find container \"83d43568034800b1b68bc6a3ab5240ed624642f1575b78ffada1ee00ddd96efd\": container with ID starting with 83d43568034800b1b68bc6a3ab5240ed624642f1575b78ffada1ee00ddd96efd not found: ID does not exist"
Jul 10 19:00:40.008996 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:40.008975    2561 scope.go:117] "RemoveContainer" containerID="7c22487a5f60f9f374595a29349ffe5253a49b643b02e64b755241a10b75d179"
Jul 10 19:00:40.009632 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:00:40.009612    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c22487a5f60f9f374595a29349ffe5253a49b643b02e64b755241a10b75d179\": container with ID starting with 7c22487a5f60f9f374595a29349ffe5253a49b643b02e64b755241a10b75d179 not found: ID does not exist" containerID="7c22487a5f60f9f374595a29349ffe5253a49b643b02e64b755241a10b75d179"
Jul 10 19:00:40.009632 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:40.009637    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c22487a5f60f9f374595a29349ffe5253a49b643b02e64b755241a10b75d179"} err="failed to get container status \"7c22487a5f60f9f374595a29349ffe5253a49b643b02e64b755241a10b75d179\": rpc error: code = NotFound desc = could not find container \"7c22487a5f60f9f374595a29349ffe5253a49b643b02e64b755241a10b75d179\": container with ID starting with 7c22487a5f60f9f374595a29349ffe5253a49b643b02e64b755241a10b75d179 not found: ID does not exist"
Jul 10 19:00:40.010402 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:40.010384    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"]
Jul 10 19:00:40.016818 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:40.016797    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b"]
Jul 10 19:00:41.078295 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:41.078262    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8744787d-10c1-476a-8494-856ac64893a9" path="/var/lib/kubelet/pods/8744787d-10c1-476a-8494-856ac64893a9/volumes"
Jul 10 19:00:42.231879 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:42.231838    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:48.932701 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:00:48.932674    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:01:06.702625 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:06.702584    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:14.348438 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:14.348402    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"]
Jul 10 19:01:14.349047 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:14.348634    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg" podUID="dc33c17a-67ba-451e-8191-79bf71bbfebb" containerName="manager" containerID="cri-o://46ff2169ebd7999c2e8876784f89efbb6f6834216b5040324416b982d708087b" gracePeriod=30
Jul 10 19:01:14.601143 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:14.601067    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"
Jul 10 19:01:14.654650 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:14.654612    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc33c17a-67ba-451e-8191-79bf71bbfebb-cert\") pod \"dc33c17a-67ba-451e-8191-79bf71bbfebb\" (UID: \"dc33c17a-67ba-451e-8191-79bf71bbfebb\") "
Jul 10 19:01:14.654846 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:14.654691    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkkrp\" (UniqueName: \"kubernetes.io/projected/dc33c17a-67ba-451e-8191-79bf71bbfebb-kube-api-access-jkkrp\") pod \"dc33c17a-67ba-451e-8191-79bf71bbfebb\" (UID: \"dc33c17a-67ba-451e-8191-79bf71bbfebb\") "
Jul 10 19:01:14.656791 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:14.656758    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc33c17a-67ba-451e-8191-79bf71bbfebb-cert" (OuterVolumeSpecName: "cert") pod "dc33c17a-67ba-451e-8191-79bf71bbfebb" (UID: "dc33c17a-67ba-451e-8191-79bf71bbfebb"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:01:14.656791 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:14.656773    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc33c17a-67ba-451e-8191-79bf71bbfebb-kube-api-access-jkkrp" (OuterVolumeSpecName: "kube-api-access-jkkrp") pod "dc33c17a-67ba-451e-8191-79bf71bbfebb" (UID: "dc33c17a-67ba-451e-8191-79bf71bbfebb"). InnerVolumeSpecName "kube-api-access-jkkrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:01:14.755331 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:14.755296    2561 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc33c17a-67ba-451e-8191-79bf71bbfebb-cert\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:01:14.755331 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:14.755325    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkkrp\" (UniqueName: \"kubernetes.io/projected/dc33c17a-67ba-451e-8191-79bf71bbfebb-kube-api-access-jkkrp\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:01:15.148477 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:15.148442    2561 generic.go:356] "Generic (PLEG): container finished" podID="dc33c17a-67ba-451e-8191-79bf71bbfebb" containerID="46ff2169ebd7999c2e8876784f89efbb6f6834216b5040324416b982d708087b" exitCode=0
Jul 10 19:01:15.148683 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:15.148486    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"
Jul 10 19:01:15.148683 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:15.148499    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg" event={"ID":"dc33c17a-67ba-451e-8191-79bf71bbfebb","Type":"ContainerDied","Data":"46ff2169ebd7999c2e8876784f89efbb6f6834216b5040324416b982d708087b"}
Jul 10 19:01:15.148683 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:15.148549    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg" event={"ID":"dc33c17a-67ba-451e-8191-79bf71bbfebb","Type":"ContainerDied","Data":"29b6cfaeec892add525cb775ee03d2d6fc1269271a8e959a73aecbc5541fa375"}
Jul 10 19:01:15.148683 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:15.148578    2561 scope.go:117] "RemoveContainer" containerID="46ff2169ebd7999c2e8876784f89efbb6f6834216b5040324416b982d708087b"
Jul 10 19:01:15.158056 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:15.158033    2561 scope.go:117] "RemoveContainer" containerID="46ff2169ebd7999c2e8876784f89efbb6f6834216b5040324416b982d708087b"
Jul 10 19:01:15.158334 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:01:15.158312    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ff2169ebd7999c2e8876784f89efbb6f6834216b5040324416b982d708087b\": container with ID starting with 46ff2169ebd7999c2e8876784f89efbb6f6834216b5040324416b982d708087b not found: ID does not exist" containerID="46ff2169ebd7999c2e8876784f89efbb6f6834216b5040324416b982d708087b"
Jul 10 19:01:15.158388 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:15.158344    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ff2169ebd7999c2e8876784f89efbb6f6834216b5040324416b982d708087b"} err="failed to get container status \"46ff2169ebd7999c2e8876784f89efbb6f6834216b5040324416b982d708087b\": rpc error: code = NotFound desc = could not find container \"46ff2169ebd7999c2e8876784f89efbb6f6834216b5040324416b982d708087b\": container with ID starting with 46ff2169ebd7999c2e8876784f89efbb6f6834216b5040324416b982d708087b not found: ID does not exist"
Jul 10 19:01:15.170651 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:15.170620    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"]
Jul 10 19:01:15.175070 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:15.175042    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-68d9dc5d5-jvjjg"]
Jul 10 19:01:17.073131 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:17.073098    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc33c17a-67ba-451e-8191-79bf71bbfebb" path="/var/lib/kubelet/pods/dc33c17a-67ba-451e-8191-79bf71bbfebb/volumes"
Jul 10 19:01:51.409624 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:51.409591    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:51.500935 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:51.500901    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:54.813184 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:01:54.813149    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:02:22.006493 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.006456    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:02:22.531742 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.531706    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"]
Jul 10 19:02:22.532152 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.532137    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8744787d-10c1-476a-8494-856ac64893a9" containerName="main"
Jul 10 19:02:22.532207 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.532153    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8744787d-10c1-476a-8494-856ac64893a9" containerName="main"
Jul 10 19:02:22.532207 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.532181    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8744787d-10c1-476a-8494-856ac64893a9" containerName="tokenizer"
Jul 10 19:02:22.532207 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.532187    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8744787d-10c1-476a-8494-856ac64893a9" containerName="tokenizer"
Jul 10 19:02:22.532207 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.532195    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8744787d-10c1-476a-8494-856ac64893a9" containerName="storage-initializer"
Jul 10 19:02:22.532207 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.532201    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8744787d-10c1-476a-8494-856ac64893a9" containerName="storage-initializer"
Jul 10 19:02:22.532207 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.532207    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc33c17a-67ba-451e-8191-79bf71bbfebb" containerName="manager"
Jul 10 19:02:22.532388 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.532212    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc33c17a-67ba-451e-8191-79bf71bbfebb" containerName="manager"
Jul 10 19:02:22.532388 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.532269    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8744787d-10c1-476a-8494-856ac64893a9" containerName="main"
Jul 10 19:02:22.532388 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.532275    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc33c17a-67ba-451e-8191-79bf71bbfebb" containerName="manager"
Jul 10 19:02:22.532388 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.532282    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8744787d-10c1-476a-8494-856ac64893a9" containerName="tokenizer"
Jul 10 19:02:22.535714 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.535692    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.542106 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.542065    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 10 19:02:22.542244 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.542065    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-g4xlw\""
Jul 10 19:02:22.565823 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.565788    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"]
Jul 10 19:02:22.655879 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.655841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.656060 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.655886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.656060 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.655941    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b160440d-93a9-4648-9dc0-f1501398f2de-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.656060 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.655973    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.656060 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.655988    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg879\" (UniqueName: \"kubernetes.io/projected/b160440d-93a9-4648-9dc0-f1501398f2de-kube-api-access-vg879\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.656060 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.656014    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.680199 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.680170    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:02:22.757382 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.757344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.757382 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.757380    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg879\" (UniqueName: \"kubernetes.io/projected/b160440d-93a9-4648-9dc0-f1501398f2de-kube-api-access-vg879\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.757597 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.757413    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.757597 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.757458    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.757597 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.757485    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.757597 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.757516    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b160440d-93a9-4648-9dc0-f1501398f2de-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.757854 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.757831    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.757890 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.757847    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.757890 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.757872    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.757957 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.757925    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.759965 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.759949    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b160440d-93a9-4648-9dc0-f1501398f2de-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.773239 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.773205    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg879\" (UniqueName: \"kubernetes.io/projected/b160440d-93a9-4648-9dc0-f1501398f2de-kube-api-access-vg879\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.846167 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.846048    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:22.990884 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:22.990857    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"]
Jul 10 19:02:22.992883 ip-10-0-135-91 kubenswrapper[2561]: W0710 19:02:22.992855    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb160440d_93a9_4648_9dc0_f1501398f2de.slice/crio-fd1d4c937fcec9408801f6228089b18717501d837161cd036e5a19202a621719 WatchSource:0}: Error finding container fd1d4c937fcec9408801f6228089b18717501d837161cd036e5a19202a621719: Status 404 returned error can't find the container with id fd1d4c937fcec9408801f6228089b18717501d837161cd036e5a19202a621719
Jul 10 19:02:23.477618 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:23.477579    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v" event={"ID":"b160440d-93a9-4648-9dc0-f1501398f2de","Type":"ContainerStarted","Data":"05c2431f2a014a5e41ec8f26b24f6c37a1cb404d06a94e4078bf2c985f569d6a"}
Jul 10 19:02:23.478012 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:23.477670    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v" event={"ID":"b160440d-93a9-4648-9dc0-f1501398f2de","Type":"ContainerStarted","Data":"fd1d4c937fcec9408801f6228089b18717501d837161cd036e5a19202a621719"}
Jul 10 19:02:24.484363 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:24.484329    2561 generic.go:356] "Generic (PLEG): container finished" podID="b160440d-93a9-4648-9dc0-f1501398f2de" containerID="05c2431f2a014a5e41ec8f26b24f6c37a1cb404d06a94e4078bf2c985f569d6a" exitCode=0
Jul 10 19:02:24.484750 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:24.484413    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v" event={"ID":"b160440d-93a9-4648-9dc0-f1501398f2de","Type":"ContainerDied","Data":"05c2431f2a014a5e41ec8f26b24f6c37a1cb404d06a94e4078bf2c985f569d6a"}
Jul 10 19:02:25.491704 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:25.491665    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v" event={"ID":"b160440d-93a9-4648-9dc0-f1501398f2de","Type":"ContainerStarted","Data":"ce9a306644fd47f29b1e42337a3584fa1215bbab7e6295d672d0f39b6a754fab"}
Jul 10 19:02:25.492154 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:25.491742    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:25.492154 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:25.491765    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v" event={"ID":"b160440d-93a9-4648-9dc0-f1501398f2de","Type":"ContainerStarted","Data":"f5ceb03b7016db518622f2639d1ab503da991902fa11984538c60e666af9e748"}
Jul 10 19:02:25.536736 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:25.536678    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v" podStartSLOduration=2.838196584 podStartE2EDuration="3.536660523s" totalImagesPullingTime="0s" totalInitContainerRuntime="698.463939ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:02:25.534931127 +0000 UTC m=+1369.009024036" watchObservedRunningTime="2026-07-10 19:02:25.536660523 +0000 UTC m=+1369.010753453"
Jul 10 19:02:32.847090 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:32.847033    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:32.847546 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:32.847115    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:32.849741 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:32.849718    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:33.532430 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:33.532399    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:02:54.538183 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:02:54.538148    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:03:02.853095 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:02.853029    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:03:02.864486 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:02.864451    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"]
Jul 10 19:03:02.864756 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:02.864722    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m" podUID="5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" containerName="main" containerID="cri-o://89d17495ec092252ca0eee6f701e7c2c66e33756440cb0c40afd23f6367c2bcf" gracePeriod=30
Jul 10 19:03:02.864756 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:02.864745    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m" podUID="5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" containerName="tokenizer" containerID="cri-o://40c4af77b06b4c108796e387e5404e2894104ff96c853302e88b673110543063" gracePeriod=30
Jul 10 19:03:03.674168 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:03.674134    2561 generic.go:356] "Generic (PLEG): container finished" podID="5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" containerID="89d17495ec092252ca0eee6f701e7c2c66e33756440cb0c40afd23f6367c2bcf" exitCode=0
Jul 10 19:03:03.674349 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:03.674194    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m" event={"ID":"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d","Type":"ContainerDied","Data":"89d17495ec092252ca0eee6f701e7c2c66e33756440cb0c40afd23f6367c2bcf"}
Jul 10 19:03:03.946062 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:03.945964    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:03:04.232258 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.232225    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:03:04.336958 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.336917    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bptc9\" (UniqueName: \"kubernetes.io/projected/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-kube-api-access-bptc9\") pod \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") "
Jul 10 19:03:04.337206 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.336973    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tls-certs\") pod \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") "
Jul 10 19:03:04.337206 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.336992    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tokenizer-cache\") pod \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") "
Jul 10 19:03:04.337206 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.337012    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tokenizer-tmp\") pod \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") "
Jul 10 19:03:04.337206 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.337030    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tokenizer-uds\") pod \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") "
Jul 10 19:03:04.337206 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.337057    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-kserve-provision-location\") pod \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\" (UID: \"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d\") "
Jul 10 19:03:04.337474 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.337369    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" (UID: "5bd1ba36-ae63-4416-9a5c-0d2715af5b7d"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:03:04.337474 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.337361    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" (UID: "5bd1ba36-ae63-4416-9a5c-0d2715af5b7d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:03:04.337474 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.337456    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" (UID: "5bd1ba36-ae63-4416-9a5c-0d2715af5b7d"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:03:04.337834 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.337811    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" (UID: "5bd1ba36-ae63-4416-9a5c-0d2715af5b7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:03:04.339274 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.339251    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-kube-api-access-bptc9" (OuterVolumeSpecName: "kube-api-access-bptc9") pod "5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" (UID: "5bd1ba36-ae63-4416-9a5c-0d2715af5b7d"). InnerVolumeSpecName "kube-api-access-bptc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:03:04.339396 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.339375    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" (UID: "5bd1ba36-ae63-4416-9a5c-0d2715af5b7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:03:04.438637 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.438584    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bptc9\" (UniqueName: \"kubernetes.io/projected/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-kube-api-access-bptc9\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:03:04.438637 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.438627    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:03:04.438637 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.438641    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tokenizer-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:03:04.438637 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.438648    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tokenizer-tmp\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:03:04.438637 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.438657    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-tokenizer-uds\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:03:04.438949 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.438664    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d-kserve-provision-location\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:03:04.681070 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.681035    2561 generic.go:356] "Generic (PLEG): container finished" podID="5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" containerID="40c4af77b06b4c108796e387e5404e2894104ff96c853302e88b673110543063" exitCode=0
Jul 10 19:03:04.681257 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.681117    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"
Jul 10 19:03:04.681257 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.681120    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m" event={"ID":"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d","Type":"ContainerDied","Data":"40c4af77b06b4c108796e387e5404e2894104ff96c853302e88b673110543063"}
Jul 10 19:03:04.681257 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.681173    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m" event={"ID":"5bd1ba36-ae63-4416-9a5c-0d2715af5b7d","Type":"ContainerDied","Data":"0398ac3d70942cccd317e31cc0d0011475db9a8bee2c8ed40e1c165e95f78e75"}
Jul 10 19:03:04.681257 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.681187    2561 scope.go:117] "RemoveContainer" containerID="40c4af77b06b4c108796e387e5404e2894104ff96c853302e88b673110543063"
Jul 10 19:03:04.690352 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.690337    2561 scope.go:117] "RemoveContainer" containerID="89d17495ec092252ca0eee6f701e7c2c66e33756440cb0c40afd23f6367c2bcf"
Jul 10 19:03:04.698732 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.698712    2561 scope.go:117] "RemoveContainer" containerID="afa70468b99adcddc7c8ebbf14e7d459ee7767bf41f0f83583e7d46379bd9068"
Jul 10 19:03:04.704483 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.704456    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"]
Jul 10 19:03:04.707234 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.707153    2561 scope.go:117] "RemoveContainer" containerID="40c4af77b06b4c108796e387e5404e2894104ff96c853302e88b673110543063"
Jul 10 19:03:04.707489 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:03:04.707469    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c4af77b06b4c108796e387e5404e2894104ff96c853302e88b673110543063\": container with ID starting with 40c4af77b06b4c108796e387e5404e2894104ff96c853302e88b673110543063 not found: ID does not exist" containerID="40c4af77b06b4c108796e387e5404e2894104ff96c853302e88b673110543063"
Jul 10 19:03:04.707584 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.707497    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c4af77b06b4c108796e387e5404e2894104ff96c853302e88b673110543063"} err="failed to get container status \"40c4af77b06b4c108796e387e5404e2894104ff96c853302e88b673110543063\": rpc error: code = NotFound desc = could not find container \"40c4af77b06b4c108796e387e5404e2894104ff96c853302e88b673110543063\": container with ID starting with 40c4af77b06b4c108796e387e5404e2894104ff96c853302e88b673110543063 not found: ID does not exist"
Jul 10 19:03:04.707584 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.707516    2561 scope.go:117] "RemoveContainer" containerID="89d17495ec092252ca0eee6f701e7c2c66e33756440cb0c40afd23f6367c2bcf"
Jul 10 19:03:04.707775 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:03:04.707756    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d17495ec092252ca0eee6f701e7c2c66e33756440cb0c40afd23f6367c2bcf\": container with ID starting with 89d17495ec092252ca0eee6f701e7c2c66e33756440cb0c40afd23f6367c2bcf not found: ID does not exist" containerID="89d17495ec092252ca0eee6f701e7c2c66e33756440cb0c40afd23f6367c2bcf"
Jul 10 19:03:04.707837 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.707780    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d17495ec092252ca0eee6f701e7c2c66e33756440cb0c40afd23f6367c2bcf"} err="failed to get container status \"89d17495ec092252ca0eee6f701e7c2c66e33756440cb0c40afd23f6367c2bcf\": rpc error: code = NotFound desc = could not find container \"89d17495ec092252ca0eee6f701e7c2c66e33756440cb0c40afd23f6367c2bcf\": container with ID starting with 89d17495ec092252ca0eee6f701e7c2c66e33756440cb0c40afd23f6367c2bcf not found: ID does not exist"
Jul 10 19:03:04.707837 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.707796    2561 scope.go:117] "RemoveContainer" containerID="afa70468b99adcddc7c8ebbf14e7d459ee7767bf41f0f83583e7d46379bd9068"
Jul 10 19:03:04.708033 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:03:04.708014    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa70468b99adcddc7c8ebbf14e7d459ee7767bf41f0f83583e7d46379bd9068\": container with ID starting with afa70468b99adcddc7c8ebbf14e7d459ee7767bf41f0f83583e7d46379bd9068 not found: ID does not exist" containerID="afa70468b99adcddc7c8ebbf14e7d459ee7767bf41f0f83583e7d46379bd9068"
Jul 10 19:03:04.708192 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.708037    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa70468b99adcddc7c8ebbf14e7d459ee7767bf41f0f83583e7d46379bd9068"} err="failed to get container status \"afa70468b99adcddc7c8ebbf14e7d459ee7767bf41f0f83583e7d46379bd9068\": rpc error: code = NotFound desc = could not find container \"afa70468b99adcddc7c8ebbf14e7d459ee7767bf41f0f83583e7d46379bd9068\": container with ID starting with afa70468b99adcddc7c8ebbf14e7d459ee7767bf41f0f83583e7d46379bd9068 not found: ID does not exist"
Jul 10 19:03:04.709232 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:04.709208    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m"]
Jul 10 19:03:05.072991 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:05.072908    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" path="/var/lib/kubelet/pods/5bd1ba36-ae63-4416-9a5c-0d2715af5b7d/volumes"
Jul 10 19:03:10.736220 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:10.736182    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:03:11.202220 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.202186    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"]
Jul 10 19:03:11.202581 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.202569    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" containerName="storage-initializer"
Jul 10 19:03:11.202628 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.202583    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" containerName="storage-initializer"
Jul 10 19:03:11.202628 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.202593    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" containerName="main"
Jul 10 19:03:11.202628 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.202598    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" containerName="main"
Jul 10 19:03:11.202628 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.202606    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" containerName="tokenizer"
Jul 10 19:03:11.202628 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.202612    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" containerName="tokenizer"
Jul 10 19:03:11.202777 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.202686    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" containerName="main"
Jul 10 19:03:11.202777 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.202695    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd1ba36-ae63-4416-9a5c-0d2715af5b7d" containerName="tokenizer"
Jul 10 19:03:11.206113 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.206097    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.208836 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.208817    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-v9vb6\""
Jul 10 19:03:11.208958 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.208818    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 10 19:03:11.217437 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.217413    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"]
Jul 10 19:03:11.298361 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.298327    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh8n9\" (UniqueName: \"kubernetes.io/projected/093aa7e9-afff-4265-84b8-7b451f45cf6b-kube-api-access-zh8n9\") pod \"router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.298569 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.298368    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093aa7e9-afff-4265-84b8-7b451f45cf6b-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.298569 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.298457    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.298569 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.298536    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.298706 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.298588    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.298706 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.298665    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.299022 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.298991    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:03:11.399979 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.399941    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.400179 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.400000    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8n9\" (UniqueName: \"kubernetes.io/projected/093aa7e9-afff-4265-84b8-7b451f45cf6b-kube-api-access-zh8n9\") pod \"router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.400179 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.400027    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093aa7e9-afff-4265-84b8-7b451f45cf6b-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.400179 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.400059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.400179 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.400117    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.400179 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.400161    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.400479 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.400459    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.400546 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.400471    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.400546 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.400501    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.400641 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.400550    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.402529 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.402511    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093aa7e9-afff-4265-84b8-7b451f45cf6b-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.410867 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.410840    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh8n9\" (UniqueName: \"kubernetes.io/projected/093aa7e9-afff-4265-84b8-7b451f45cf6b-kube-api-access-zh8n9\") pod \"router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.515471 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.515372    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:11.683034 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.682977    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"]
Jul 10 19:03:11.685656 ip-10-0-135-91 kubenswrapper[2561]: W0710 19:03:11.685618    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093aa7e9_afff_4265_84b8_7b451f45cf6b.slice/crio-acd89baa44d9f9246795b5a35e510dcc2c4f9a9c03c6d0277f547c873eaa99fd WatchSource:0}: Error finding container acd89baa44d9f9246795b5a35e510dcc2c4f9a9c03c6d0277f547c873eaa99fd: Status 404 returned error can't find the container with id acd89baa44d9f9246795b5a35e510dcc2c4f9a9c03c6d0277f547c873eaa99fd
Jul 10 19:03:11.716756 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:11.716724    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx" event={"ID":"093aa7e9-afff-4265-84b8-7b451f45cf6b","Type":"ContainerStarted","Data":"acd89baa44d9f9246795b5a35e510dcc2c4f9a9c03c6d0277f547c873eaa99fd"}
Jul 10 19:03:12.725683 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:12.725642    2561 generic.go:356] "Generic (PLEG): container finished" podID="093aa7e9-afff-4265-84b8-7b451f45cf6b" containerID="d0a6effddc88064e51498ce8d443bf72bf988b74c7ac44e6842a9a9888acc282" exitCode=0
Jul 10 19:03:12.726129 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:12.725689    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx" event={"ID":"093aa7e9-afff-4265-84b8-7b451f45cf6b","Type":"ContainerDied","Data":"d0a6effddc88064e51498ce8d443bf72bf988b74c7ac44e6842a9a9888acc282"}
Jul 10 19:03:13.732750 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:13.732716    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx" event={"ID":"093aa7e9-afff-4265-84b8-7b451f45cf6b","Type":"ContainerStarted","Data":"b9490db86c5099f8f2c172b8f65317516ab8874f8123e93444e6945a70c69696"}
Jul 10 19:03:13.733179 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:13.732767    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx" event={"ID":"093aa7e9-afff-4265-84b8-7b451f45cf6b","Type":"ContainerStarted","Data":"c53ecd885bb387fff1c5dc31ed9505fb3dd78b560a2983a7bb42319b6752b1c1"}
Jul 10 19:03:13.733179 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:13.732850    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:21.515491 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:21.515458    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:21.515928 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:21.515530    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:21.518699 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:21.518672    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:21.546488 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:21.546436    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx" podStartSLOduration=9.800213725 podStartE2EDuration="10.546420518s" totalImagesPullingTime="0s" totalInitContainerRuntime="746.206793ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:03:13.759190688 +0000 UTC m=+1417.233283574" watchObservedRunningTime="2026-07-10 19:03:21.546420518 +0000 UTC m=+1425.020513402"
Jul 10 19:03:21.774399 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:21.774319    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:03:42.779517 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:03:42.779486    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:04:37.095834 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:04:37.095804    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 19:04:37.096590 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:04:37.096572    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 19:04:37.102942 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:04:37.102919    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 19:04:37.103570 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:04:37.103547    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 19:06:02.134872 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:06:02.134833    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:06:21.637672 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:06:21.637632    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:07:48.033285 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:07:48.033252    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:07:48.133945 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:07:48.133910    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:09:37.130767 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:09:37.130730    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 19:09:37.131718 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:09:37.131697    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 19:09:37.138310 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:09:37.138288    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 19:09:37.139008 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:09:37.138986    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 19:12:26.100791 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:12:26.100756    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:14:37.164420 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:14:37.164387    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 19:14:37.166251 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:14:37.166227    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 19:14:37.172421 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:14:37.172393    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 19:14:37.173846 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:14:37.173826    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 19:14:44.092676 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:14:44.092635    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:15:08.486526 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:15:08.486489    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:16:55.425978 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:55.425943    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"]
Jul 10 19:16:55.426622 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:55.426212    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v" podUID="b160440d-93a9-4648-9dc0-f1501398f2de" containerName="main" containerID="cri-o://f5ceb03b7016db518622f2639d1ab503da991902fa11984538c60e666af9e748" gracePeriod=30
Jul 10 19:16:55.426622 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:55.426285    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v" podUID="b160440d-93a9-4648-9dc0-f1501398f2de" containerName="tokenizer" containerID="cri-o://ce9a306644fd47f29b1e42337a3584fa1215bbab7e6295d672d0f39b6a754fab" gracePeriod=30
Jul 10 19:16:55.447278 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:55.447251    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:16:55.510257 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:55.510215    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:16:55.719168 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:55.719050    2561 generic.go:356] "Generic (PLEG): container finished" podID="b160440d-93a9-4648-9dc0-f1501398f2de" containerID="f5ceb03b7016db518622f2639d1ab503da991902fa11984538c60e666af9e748" exitCode=0
Jul 10 19:16:55.719168 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:55.719116    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v" event={"ID":"b160440d-93a9-4648-9dc0-f1501398f2de","Type":"ContainerDied","Data":"f5ceb03b7016db518622f2639d1ab503da991902fa11984538c60e666af9e748"}
Jul 10 19:16:56.726443 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.726410    2561 generic.go:356] "Generic (PLEG): container finished" podID="b160440d-93a9-4648-9dc0-f1501398f2de" containerID="ce9a306644fd47f29b1e42337a3584fa1215bbab7e6295d672d0f39b6a754fab" exitCode=0
Jul 10 19:16:56.726800 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.726484    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v" event={"ID":"b160440d-93a9-4648-9dc0-f1501398f2de","Type":"ContainerDied","Data":"ce9a306644fd47f29b1e42337a3584fa1215bbab7e6295d672d0f39b6a754fab"}
Jul 10 19:16:56.789581 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.789556    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:16:56.809476 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.809444    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg879\" (UniqueName: \"kubernetes.io/projected/b160440d-93a9-4648-9dc0-f1501398f2de-kube-api-access-vg879\") pod \"b160440d-93a9-4648-9dc0-f1501398f2de\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") "
Jul 10 19:16:56.809647 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.809496    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-tokenizer-uds\") pod \"b160440d-93a9-4648-9dc0-f1501398f2de\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") "
Jul 10 19:16:56.809647 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.809520    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-tokenizer-tmp\") pod \"b160440d-93a9-4648-9dc0-f1501398f2de\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") "
Jul 10 19:16:56.809647 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.809579    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b160440d-93a9-4648-9dc0-f1501398f2de-tls-certs\") pod \"b160440d-93a9-4648-9dc0-f1501398f2de\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") "
Jul 10 19:16:56.809818 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.809646    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-kserve-provision-location\") pod \"b160440d-93a9-4648-9dc0-f1501398f2de\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") "
Jul 10 19:16:56.809818 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.809678    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-tokenizer-cache\") pod \"b160440d-93a9-4648-9dc0-f1501398f2de\" (UID: \"b160440d-93a9-4648-9dc0-f1501398f2de\") "
Jul 10 19:16:56.809919 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.809831    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b160440d-93a9-4648-9dc0-f1501398f2de" (UID: "b160440d-93a9-4648-9dc0-f1501398f2de"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:16:56.809971 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.809928    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "b160440d-93a9-4648-9dc0-f1501398f2de" (UID: "b160440d-93a9-4648-9dc0-f1501398f2de"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:16:56.810027 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.809983    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-tokenizer-uds\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:16:56.810027 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.810003    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "b160440d-93a9-4648-9dc0-f1501398f2de" (UID: "b160440d-93a9-4648-9dc0-f1501398f2de"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:16:56.810404 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.810382    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b160440d-93a9-4648-9dc0-f1501398f2de" (UID: "b160440d-93a9-4648-9dc0-f1501398f2de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:16:56.811756 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.811729    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b160440d-93a9-4648-9dc0-f1501398f2de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b160440d-93a9-4648-9dc0-f1501398f2de" (UID: "b160440d-93a9-4648-9dc0-f1501398f2de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:16:56.811756 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.811743    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b160440d-93a9-4648-9dc0-f1501398f2de-kube-api-access-vg879" (OuterVolumeSpecName: "kube-api-access-vg879") pod "b160440d-93a9-4648-9dc0-f1501398f2de" (UID: "b160440d-93a9-4648-9dc0-f1501398f2de"). InnerVolumeSpecName "kube-api-access-vg879". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:16:56.910808 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.910771    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-kserve-provision-location\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:16:56.910808 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.910806    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-tokenizer-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:16:56.910808 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.910815    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg879\" (UniqueName: \"kubernetes.io/projected/b160440d-93a9-4648-9dc0-f1501398f2de-kube-api-access-vg879\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:16:56.911045 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.910824    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b160440d-93a9-4648-9dc0-f1501398f2de-tokenizer-tmp\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:16:56.911045 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:56.910833    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b160440d-93a9-4648-9dc0-f1501398f2de-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:16:57.733226 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:57.733196    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"
Jul 10 19:16:57.733226 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:57.733208    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v" event={"ID":"b160440d-93a9-4648-9dc0-f1501398f2de","Type":"ContainerDied","Data":"fd1d4c937fcec9408801f6228089b18717501d837161cd036e5a19202a621719"}
Jul 10 19:16:57.733722 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:57.733265    2561 scope.go:117] "RemoveContainer" containerID="ce9a306644fd47f29b1e42337a3584fa1215bbab7e6295d672d0f39b6a754fab"
Jul 10 19:16:57.742764 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:57.742746    2561 scope.go:117] "RemoveContainer" containerID="f5ceb03b7016db518622f2639d1ab503da991902fa11984538c60e666af9e748"
Jul 10 19:16:57.751018 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:57.750958    2561 scope.go:117] "RemoveContainer" containerID="05c2431f2a014a5e41ec8f26b24f6c37a1cb404d06a94e4078bf2c985f569d6a"
Jul 10 19:16:57.752757 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:57.752733    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"]
Jul 10 19:16:57.759162 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:57.759131    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v"]
Jul 10 19:16:58.700594 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:58.700556    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:16:59.072229 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:16:59.072143    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b160440d-93a9-4648-9dc0-f1501398f2de" path="/var/lib/kubelet/pods/b160440d-93a9-4648-9dc0-f1501398f2de/volumes"
Jul 10 19:17:34.980635 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:34.980600    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"]
Jul 10 19:17:34.983667 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:34.980941    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx" podUID="093aa7e9-afff-4265-84b8-7b451f45cf6b" containerName="main" containerID="cri-o://c53ecd885bb387fff1c5dc31ed9505fb3dd78b560a2983a7bb42319b6752b1c1" gracePeriod=30
Jul 10 19:17:34.983667 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:34.981032    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx" podUID="093aa7e9-afff-4265-84b8-7b451f45cf6b" containerName="tokenizer" containerID="cri-o://b9490db86c5099f8f2c172b8f65317516ab8874f8123e93444e6945a70c69696" gracePeriod=30
Jul 10 19:17:35.011274 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:35.011247    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:17:35.915040 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:35.915004    2561 generic.go:356] "Generic (PLEG): container finished" podID="093aa7e9-afff-4265-84b8-7b451f45cf6b" containerID="c53ecd885bb387fff1c5dc31ed9505fb3dd78b560a2983a7bb42319b6752b1c1" exitCode=0
Jul 10 19:17:35.915223 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:35.915069    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx" event={"ID":"093aa7e9-afff-4265-84b8-7b451f45cf6b","Type":"ContainerDied","Data":"c53ecd885bb387fff1c5dc31ed9505fb3dd78b560a2983a7bb42319b6752b1c1"}
Jul 10 19:17:36.237859 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.237835    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:17:36.364164 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.364129    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-tokenizer-uds\") pod \"093aa7e9-afff-4265-84b8-7b451f45cf6b\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") "
Jul 10 19:17:36.364164 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.364167    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-tokenizer-tmp\") pod \"093aa7e9-afff-4265-84b8-7b451f45cf6b\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") "
Jul 10 19:17:36.364391 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.364248    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-tokenizer-cache\") pod \"093aa7e9-afff-4265-84b8-7b451f45cf6b\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") "
Jul 10 19:17:36.364391 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.364298    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093aa7e9-afff-4265-84b8-7b451f45cf6b-tls-certs\") pod \"093aa7e9-afff-4265-84b8-7b451f45cf6b\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") "
Jul 10 19:17:36.364391 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.364325    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-kserve-provision-location\") pod \"093aa7e9-afff-4265-84b8-7b451f45cf6b\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") "
Jul 10 19:17:36.364391 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.364360    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh8n9\" (UniqueName: \"kubernetes.io/projected/093aa7e9-afff-4265-84b8-7b451f45cf6b-kube-api-access-zh8n9\") pod \"093aa7e9-afff-4265-84b8-7b451f45cf6b\" (UID: \"093aa7e9-afff-4265-84b8-7b451f45cf6b\") "
Jul 10 19:17:36.364576 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.364474    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "093aa7e9-afff-4265-84b8-7b451f45cf6b" (UID: "093aa7e9-afff-4265-84b8-7b451f45cf6b"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:17:36.364576 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.364541    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "093aa7e9-afff-4265-84b8-7b451f45cf6b" (UID: "093aa7e9-afff-4265-84b8-7b451f45cf6b"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:17:36.364689 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.364605    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "093aa7e9-afff-4265-84b8-7b451f45cf6b" (UID: "093aa7e9-afff-4265-84b8-7b451f45cf6b"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:17:36.364746 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.364696    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-tokenizer-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:36.364746 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.364715    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-tokenizer-uds\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:36.364746 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.364728    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-tokenizer-tmp\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:36.365114 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.365067    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "093aa7e9-afff-4265-84b8-7b451f45cf6b" (UID: "093aa7e9-afff-4265-84b8-7b451f45cf6b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:17:36.366438 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.366412    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/093aa7e9-afff-4265-84b8-7b451f45cf6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "093aa7e9-afff-4265-84b8-7b451f45cf6b" (UID: "093aa7e9-afff-4265-84b8-7b451f45cf6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:17:36.366546 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.366522    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093aa7e9-afff-4265-84b8-7b451f45cf6b-kube-api-access-zh8n9" (OuterVolumeSpecName: "kube-api-access-zh8n9") pod "093aa7e9-afff-4265-84b8-7b451f45cf6b" (UID: "093aa7e9-afff-4265-84b8-7b451f45cf6b"). InnerVolumeSpecName "kube-api-access-zh8n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:17:36.419633 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.419601    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:17:36.465564 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.465531    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093aa7e9-afff-4265-84b8-7b451f45cf6b-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:36.465564 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.465560    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/093aa7e9-afff-4265-84b8-7b451f45cf6b-kserve-provision-location\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:36.465564 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.465572    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh8n9\" (UniqueName: \"kubernetes.io/projected/093aa7e9-afff-4265-84b8-7b451f45cf6b-kube-api-access-zh8n9\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:36.921034 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.920995    2561 generic.go:356] "Generic (PLEG): container finished" podID="093aa7e9-afff-4265-84b8-7b451f45cf6b" containerID="b9490db86c5099f8f2c172b8f65317516ab8874f8123e93444e6945a70c69696" exitCode=0
Jul 10 19:17:36.921211 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.921056    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"
Jul 10 19:17:36.921211 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.921111    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx" event={"ID":"093aa7e9-afff-4265-84b8-7b451f45cf6b","Type":"ContainerDied","Data":"b9490db86c5099f8f2c172b8f65317516ab8874f8123e93444e6945a70c69696"}
Jul 10 19:17:36.921211 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.921176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx" event={"ID":"093aa7e9-afff-4265-84b8-7b451f45cf6b","Type":"ContainerDied","Data":"acd89baa44d9f9246795b5a35e510dcc2c4f9a9c03c6d0277f547c873eaa99fd"}
Jul 10 19:17:36.921211 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.921207    2561 scope.go:117] "RemoveContainer" containerID="b9490db86c5099f8f2c172b8f65317516ab8874f8123e93444e6945a70c69696"
Jul 10 19:17:36.930919 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.930899    2561 scope.go:117] "RemoveContainer" containerID="c53ecd885bb387fff1c5dc31ed9505fb3dd78b560a2983a7bb42319b6752b1c1"
Jul 10 19:17:36.938872 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.938850    2561 scope.go:117] "RemoveContainer" containerID="d0a6effddc88064e51498ce8d443bf72bf988b74c7ac44e6842a9a9888acc282"
Jul 10 19:17:36.945383 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.945357    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"]
Jul 10 19:17:36.948044 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.948024    2561 scope.go:117] "RemoveContainer" containerID="b9490db86c5099f8f2c172b8f65317516ab8874f8123e93444e6945a70c69696"
Jul 10 19:17:36.948362 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.948342    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx"]
Jul 10 19:17:36.948420 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:17:36.948341    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9490db86c5099f8f2c172b8f65317516ab8874f8123e93444e6945a70c69696\": container with ID starting with b9490db86c5099f8f2c172b8f65317516ab8874f8123e93444e6945a70c69696 not found: ID does not exist" containerID="b9490db86c5099f8f2c172b8f65317516ab8874f8123e93444e6945a70c69696"
Jul 10 19:17:36.948420 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.948402    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9490db86c5099f8f2c172b8f65317516ab8874f8123e93444e6945a70c69696"} err="failed to get container status \"b9490db86c5099f8f2c172b8f65317516ab8874f8123e93444e6945a70c69696\": rpc error: code = NotFound desc = could not find container \"b9490db86c5099f8f2c172b8f65317516ab8874f8123e93444e6945a70c69696\": container with ID starting with b9490db86c5099f8f2c172b8f65317516ab8874f8123e93444e6945a70c69696 not found: ID does not exist"
Jul 10 19:17:36.948501 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.948419    2561 scope.go:117] "RemoveContainer" containerID="c53ecd885bb387fff1c5dc31ed9505fb3dd78b560a2983a7bb42319b6752b1c1"
Jul 10 19:17:36.948649 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:17:36.948634    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c53ecd885bb387fff1c5dc31ed9505fb3dd78b560a2983a7bb42319b6752b1c1\": container with ID starting with c53ecd885bb387fff1c5dc31ed9505fb3dd78b560a2983a7bb42319b6752b1c1 not found: ID does not exist" containerID="c53ecd885bb387fff1c5dc31ed9505fb3dd78b560a2983a7bb42319b6752b1c1"
Jul 10 19:17:36.948693 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.948653    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53ecd885bb387fff1c5dc31ed9505fb3dd78b560a2983a7bb42319b6752b1c1"} err="failed to get container status \"c53ecd885bb387fff1c5dc31ed9505fb3dd78b560a2983a7bb42319b6752b1c1\": rpc error: code = NotFound desc = could not find container \"c53ecd885bb387fff1c5dc31ed9505fb3dd78b560a2983a7bb42319b6752b1c1\": container with ID starting with c53ecd885bb387fff1c5dc31ed9505fb3dd78b560a2983a7bb42319b6752b1c1 not found: ID does not exist"
Jul 10 19:17:36.948693 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.948666    2561 scope.go:117] "RemoveContainer" containerID="d0a6effddc88064e51498ce8d443bf72bf988b74c7ac44e6842a9a9888acc282"
Jul 10 19:17:36.948920 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:17:36.948901    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0a6effddc88064e51498ce8d443bf72bf988b74c7ac44e6842a9a9888acc282\": container with ID starting with d0a6effddc88064e51498ce8d443bf72bf988b74c7ac44e6842a9a9888acc282 not found: ID does not exist" containerID="d0a6effddc88064e51498ce8d443bf72bf988b74c7ac44e6842a9a9888acc282"
Jul 10 19:17:36.948963 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:36.948927    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0a6effddc88064e51498ce8d443bf72bf988b74c7ac44e6842a9a9888acc282"} err="failed to get container status \"d0a6effddc88064e51498ce8d443bf72bf988b74c7ac44e6842a9a9888acc282\": rpc error: code = NotFound desc = could not find container \"d0a6effddc88064e51498ce8d443bf72bf988b74c7ac44e6842a9a9888acc282\": container with ID starting with d0a6effddc88064e51498ce8d443bf72bf988b74c7ac44e6842a9a9888acc282 not found: ID does not exist"
Jul 10 19:17:37.072549 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:37.072519    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="093aa7e9-afff-4265-84b8-7b451f45cf6b" path="/var/lib/kubelet/pods/093aa7e9-afff-4265-84b8-7b451f45cf6b/volumes"
Jul 10 19:17:48.083513 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.083480    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:17:48.279886 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.279855    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"]
Jul 10 19:17:48.280443 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.280426    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="093aa7e9-afff-4265-84b8-7b451f45cf6b" containerName="main"
Jul 10 19:17:48.280513 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.280446    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="093aa7e9-afff-4265-84b8-7b451f45cf6b" containerName="main"
Jul 10 19:17:48.280513 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.280470    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b160440d-93a9-4648-9dc0-f1501398f2de" containerName="storage-initializer"
Jul 10 19:17:48.280513 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.280479    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b160440d-93a9-4648-9dc0-f1501398f2de" containerName="storage-initializer"
Jul 10 19:17:48.280513 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.280494    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="093aa7e9-afff-4265-84b8-7b451f45cf6b" containerName="storage-initializer"
Jul 10 19:17:48.280513 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.280503    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="093aa7e9-afff-4265-84b8-7b451f45cf6b" containerName="storage-initializer"
Jul 10 19:17:48.280665 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.280519    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b160440d-93a9-4648-9dc0-f1501398f2de" containerName="main"
Jul 10 19:17:48.280665 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.280527    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b160440d-93a9-4648-9dc0-f1501398f2de" containerName="main"
Jul 10 19:17:48.280665 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.280538    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="093aa7e9-afff-4265-84b8-7b451f45cf6b" containerName="tokenizer"
Jul 10 19:17:48.280665 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.280546    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="093aa7e9-afff-4265-84b8-7b451f45cf6b" containerName="tokenizer"
Jul 10 19:17:48.280665 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.280572    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b160440d-93a9-4648-9dc0-f1501398f2de" containerName="tokenizer"
Jul 10 19:17:48.280665 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.280580    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b160440d-93a9-4648-9dc0-f1501398f2de" containerName="tokenizer"
Jul 10 19:17:48.280665 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.280660    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b160440d-93a9-4648-9dc0-f1501398f2de" containerName="main"
Jul 10 19:17:48.280868 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.280672    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="093aa7e9-afff-4265-84b8-7b451f45cf6b" containerName="tokenizer"
Jul 10 19:17:48.280868 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.280681    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="093aa7e9-afff-4265-84b8-7b451f45cf6b" containerName="main"
Jul 10 19:17:48.280868 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.280693    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b160440d-93a9-4648-9dc0-f1501398f2de" containerName="tokenizer"
Jul 10 19:17:48.283926 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.283903    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.286522 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.286495    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-epp-sa-dockercfg-mjc25\""
Jul 10 19:17:48.286644 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.286525    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 19:17:48.286644 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.286590    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 19:17:48.286784 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.286645    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jjdpb\""
Jul 10 19:17:48.287327 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.287307    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
Jul 10 19:17:48.298118 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.298091    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"]
Jul 10 19:17:48.326584 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.326557    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:17:48.360155 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.360038    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7wtd\" (UniqueName: \"kubernetes.io/projected/8c01643a-50e4-4e66-b131-47e6565ec206-kube-api-access-n7wtd\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.360155 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.360115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.360155 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.360158    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.360389 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.360183    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c01643a-50e4-4e66-b131-47e6565ec206-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.360389 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.360209    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.360389 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.360272    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.461642 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.461600    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7wtd\" (UniqueName: \"kubernetes.io/projected/8c01643a-50e4-4e66-b131-47e6565ec206-kube-api-access-n7wtd\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.461815 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.461647    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.461864 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.461822    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.461911 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.461872    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c01643a-50e4-4e66-b131-47e6565ec206-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.461911 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.461897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.461991 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.461968    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.462240 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.462043    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.462240 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.462209    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.462440 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.462297    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.462440 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.462364    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.464570 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.464547    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c01643a-50e4-4e66-b131-47e6565ec206-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.469371 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.469349    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7wtd\" (UniqueName: \"kubernetes.io/projected/8c01643a-50e4-4e66-b131-47e6565ec206-kube-api-access-n7wtd\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.594364 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.594325    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:48.737067 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.737034    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"]
Jul 10 19:17:48.741458 ip-10-0-135-91 kubenswrapper[2561]: W0710 19:17:48.741429    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c01643a_50e4_4e66_b131_47e6565ec206.slice/crio-b37c548c04839eb38d70fbe6e747a87835ab9beca62f7228182f6bc87c946572 WatchSource:0}: Error finding container b37c548c04839eb38d70fbe6e747a87835ab9beca62f7228182f6bc87c946572: Status 404 returned error can't find the container with id b37c548c04839eb38d70fbe6e747a87835ab9beca62f7228182f6bc87c946572
Jul 10 19:17:48.977064 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.977029    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc" event={"ID":"8c01643a-50e4-4e66-b131-47e6565ec206","Type":"ContainerStarted","Data":"6449a199825208a68733957aab5c311b076cfcd6dd31830fa98392f571bad613"}
Jul 10 19:17:48.977243 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:48.977115    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc" event={"ID":"8c01643a-50e4-4e66-b131-47e6565ec206","Type":"ContainerStarted","Data":"b37c548c04839eb38d70fbe6e747a87835ab9beca62f7228182f6bc87c946572"}
Jul 10 19:17:49.983794 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:49.983752    2561 generic.go:356] "Generic (PLEG): container finished" podID="8c01643a-50e4-4e66-b131-47e6565ec206" containerID="6449a199825208a68733957aab5c311b076cfcd6dd31830fa98392f571bad613" exitCode=0
Jul 10 19:17:49.984196 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:49.983829    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc" event={"ID":"8c01643a-50e4-4e66-b131-47e6565ec206","Type":"ContainerDied","Data":"6449a199825208a68733957aab5c311b076cfcd6dd31830fa98392f571bad613"}
Jul 10 19:17:50.992086 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:50.992035    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc" event={"ID":"8c01643a-50e4-4e66-b131-47e6565ec206","Type":"ContainerStarted","Data":"dabbd3fc70aab1484087b8268f3e6b1d7721ad277316a02978f189cd1cd2d34b"}
Jul 10 19:17:50.992505 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:50.992123    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc" event={"ID":"8c01643a-50e4-4e66-b131-47e6565ec206","Type":"ContainerStarted","Data":"eef29f1a007cf571fcc181bbe4c317e97468a535b20ab023c2a532288418547b"}
Jul 10 19:17:50.992505 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:50.992155    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:51.020380 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:51.020313    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc" podStartSLOduration=2.2696417269999998 podStartE2EDuration="3.02028917s" totalImagesPullingTime="0s" totalInitContainerRuntime="750.647443ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:17:51.018144566 +0000 UTC m=+2294.492237462" watchObservedRunningTime="2026-07-10 19:17:51.02028917 +0000 UTC m=+2294.494382058"
Jul 10 19:17:58.104378 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:58.104344    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:17:58.595019 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:58.594974    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:58.595229 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:58.595046    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:58.597792 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:58.597766    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:17:59.031889 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:17:59.031861    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:18:20.042303 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:18:20.042270    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:18:59.556613 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:18:59.556539    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:19:13.350819 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:13.350782    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:19:13.503788 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:13.503748    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:19:13.603890 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:13.603801    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:19:13.774953 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:13.774914    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"]
Jul 10 19:19:13.778622 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:13.778597    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:13.781892 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:13.781864    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
Jul 10 19:19:13.782019 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:13.781960    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5ec-epp-sa-dockercfg-lss44\""
Jul 10 19:19:13.797802 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:13.797768    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"]
Jul 10 19:19:13.835500 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:13.835457    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:19:13.905471 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:13.905434    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:13.905647 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:13.905478    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:13.905647 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:13.905513    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c8nt\" (UniqueName: \"kubernetes.io/projected/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-kube-api-access-4c8nt\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:13.905647 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:13.905591    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:13.905647 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:13.905626    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:13.905845 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:13.905710    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:14.006577 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:14.006533    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:14.006577 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:14.006578    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:14.006827 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:14.006600    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c8nt\" (UniqueName: \"kubernetes.io/projected/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-kube-api-access-4c8nt\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:14.006827 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:14.006647    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:14.006827 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:14.006671    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:14.006827 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:14.006728    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:14.007051 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:14.007028    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:14.007150 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:14.007066    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:14.007150 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:14.007035    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:14.007234 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:14.007165    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:14.009137 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:14.009119    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:14.015791 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:14.015762    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c8nt\" (UniqueName: \"kubernetes.io/projected/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-kube-api-access-4c8nt\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:14.088186 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:14.088151    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:14.244030 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:14.244002    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"]
Jul 10 19:19:14.246523 ip-10-0-135-91 kubenswrapper[2561]: W0710 19:19:14.246478    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b3c1ee_3308_4474_a844_a94e1b8d0dcd.slice/crio-ce2c3fb8e6939ca650127fee6cd6cb304eabc439d0cd40f2c8c96424af6d452c WatchSource:0}: Error finding container ce2c3fb8e6939ca650127fee6cd6cb304eabc439d0cd40f2c8c96424af6d452c: Status 404 returned error can't find the container with id ce2c3fb8e6939ca650127fee6cd6cb304eabc439d0cd40f2c8c96424af6d452c
Jul 10 19:19:14.406782 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:14.406746    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9" event={"ID":"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd","Type":"ContainerStarted","Data":"88ed31d3e939b7f9f7931d2be3ed9bdc0dc4d15b6d802b0c371c0ca7e8788e17"}
Jul 10 19:19:14.407205 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:14.406804    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9" event={"ID":"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd","Type":"ContainerStarted","Data":"ce2c3fb8e6939ca650127fee6cd6cb304eabc439d0cd40f2c8c96424af6d452c"}
Jul 10 19:19:15.413498 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:15.413461    2561 generic.go:356] "Generic (PLEG): container finished" podID="f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" containerID="88ed31d3e939b7f9f7931d2be3ed9bdc0dc4d15b6d802b0c371c0ca7e8788e17" exitCode=0
Jul 10 19:19:15.413908 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:15.413541    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9" event={"ID":"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd","Type":"ContainerDied","Data":"88ed31d3e939b7f9f7931d2be3ed9bdc0dc4d15b6d802b0c371c0ca7e8788e17"}
Jul 10 19:19:16.420854 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:16.420810    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9" event={"ID":"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd","Type":"ContainerStarted","Data":"22cf9008afd01f71cec2875b6ba6f65cd8f973fdfae31256d4317914ebd6d85a"}
Jul 10 19:19:16.421370 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:16.420878    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9" event={"ID":"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd","Type":"ContainerStarted","Data":"8b8026370fc3e193f39d9cdaa6559fcf949165b2770127366ea81c0bbd2136f8"}
Jul 10 19:19:16.421370 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:16.421005    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:16.447811 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:16.447750    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9" podStartSLOduration=2.699954681 podStartE2EDuration="3.447732659s" totalImagesPullingTime="0s" totalInitContainerRuntime="747.777978ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:19:16.44538701 +0000 UTC m=+2379.919479896" watchObservedRunningTime="2026-07-10 19:19:16.447732659 +0000 UTC m=+2379.921825541"
Jul 10 19:19:24.088870 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:24.088826    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:24.089437 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:24.088903    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:24.091795 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:24.091773    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:24.399143 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:24.398990    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:19:24.465162 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:24.465131    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:19:37.195365 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:37.195335    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 19:19:37.198572 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:37.198547    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 19:19:37.202650 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:37.202628    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 19:19:37.206228 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:37.206210    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjq8k_9168c941-ca9d-40f0-8543-f3f103f9a50f/ovn-acl-logging/0.log"
Jul 10 19:19:45.470697 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:19:45.470660    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:20:47.079156 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:47.079118    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:20:47.085025 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:47.085001    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"]
Jul 10 19:20:47.085248 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:47.085222    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc" podUID="8c01643a-50e4-4e66-b131-47e6565ec206" containerName="main" containerID="cri-o://eef29f1a007cf571fcc181bbe4c317e97468a535b20ab023c2a532288418547b" gracePeriod=30
Jul 10 19:20:47.085327 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:47.085272    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc" podUID="8c01643a-50e4-4e66-b131-47e6565ec206" containerName="tokenizer" containerID="cri-o://dabbd3fc70aab1484087b8268f3e6b1d7721ad277316a02978f189cd1cd2d34b" gracePeriod=30
Jul 10 19:20:47.173659 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:47.173621    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:20:47.860033 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:47.860003    2561 generic.go:356] "Generic (PLEG): container finished" podID="8c01643a-50e4-4e66-b131-47e6565ec206" containerID="eef29f1a007cf571fcc181bbe4c317e97468a535b20ab023c2a532288418547b" exitCode=0
Jul 10 19:20:47.860222 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:47.860088    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc" event={"ID":"8c01643a-50e4-4e66-b131-47e6565ec206","Type":"ContainerDied","Data":"eef29f1a007cf571fcc181bbe4c317e97468a535b20ab023c2a532288418547b"}
Jul 10 19:20:48.457851 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.457822    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:20:48.538876 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.538771    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7wtd\" (UniqueName: \"kubernetes.io/projected/8c01643a-50e4-4e66-b131-47e6565ec206-kube-api-access-n7wtd\") pod \"8c01643a-50e4-4e66-b131-47e6565ec206\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") "
Jul 10 19:20:48.538876 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.538837    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-tokenizer-tmp\") pod \"8c01643a-50e4-4e66-b131-47e6565ec206\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") "
Jul 10 19:20:48.538876 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.538866    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-tokenizer-uds\") pod \"8c01643a-50e4-4e66-b131-47e6565ec206\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") "
Jul 10 19:20:48.539196 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.538925    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-kserve-provision-location\") pod \"8c01643a-50e4-4e66-b131-47e6565ec206\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") "
Jul 10 19:20:48.539196 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.538968    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-tokenizer-cache\") pod \"8c01643a-50e4-4e66-b131-47e6565ec206\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") "
Jul 10 19:20:48.539196 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.539044    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c01643a-50e4-4e66-b131-47e6565ec206-tls-certs\") pod \"8c01643a-50e4-4e66-b131-47e6565ec206\" (UID: \"8c01643a-50e4-4e66-b131-47e6565ec206\") "
Jul 10 19:20:48.539335 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.539297    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8c01643a-50e4-4e66-b131-47e6565ec206" (UID: "8c01643a-50e4-4e66-b131-47e6565ec206"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:20:48.539335 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.539308    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "8c01643a-50e4-4e66-b131-47e6565ec206" (UID: "8c01643a-50e4-4e66-b131-47e6565ec206"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:20:48.539438 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.539342    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "8c01643a-50e4-4e66-b131-47e6565ec206" (UID: "8c01643a-50e4-4e66-b131-47e6565ec206"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:20:48.539486 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.539454    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-tokenizer-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:20:48.539486 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.539470    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-tokenizer-tmp\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:20:48.539486 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.539483    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-tokenizer-uds\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:20:48.539730 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.539707    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c01643a-50e4-4e66-b131-47e6565ec206" (UID: "8c01643a-50e4-4e66-b131-47e6565ec206"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:20:48.541036 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.541016    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c01643a-50e4-4e66-b131-47e6565ec206-kube-api-access-n7wtd" (OuterVolumeSpecName: "kube-api-access-n7wtd") pod "8c01643a-50e4-4e66-b131-47e6565ec206" (UID: "8c01643a-50e4-4e66-b131-47e6565ec206"). InnerVolumeSpecName "kube-api-access-n7wtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:20:48.541265 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.541242    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c01643a-50e4-4e66-b131-47e6565ec206-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c01643a-50e4-4e66-b131-47e6565ec206" (UID: "8c01643a-50e4-4e66-b131-47e6565ec206"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:20:48.639932 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.639895    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c01643a-50e4-4e66-b131-47e6565ec206-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:20:48.639932 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.639927    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7wtd\" (UniqueName: \"kubernetes.io/projected/8c01643a-50e4-4e66-b131-47e6565ec206-kube-api-access-n7wtd\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:20:48.639932 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.639937    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c01643a-50e4-4e66-b131-47e6565ec206-kserve-provision-location\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:20:48.867370 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.867279    2561 generic.go:356] "Generic (PLEG): container finished" podID="8c01643a-50e4-4e66-b131-47e6565ec206" containerID="dabbd3fc70aab1484087b8268f3e6b1d7721ad277316a02978f189cd1cd2d34b" exitCode=0
Jul 10 19:20:48.867370 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.867342    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"
Jul 10 19:20:48.867610 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.867366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc" event={"ID":"8c01643a-50e4-4e66-b131-47e6565ec206","Type":"ContainerDied","Data":"dabbd3fc70aab1484087b8268f3e6b1d7721ad277316a02978f189cd1cd2d34b"}
Jul 10 19:20:48.867610 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.867469    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc" event={"ID":"8c01643a-50e4-4e66-b131-47e6565ec206","Type":"ContainerDied","Data":"b37c548c04839eb38d70fbe6e747a87835ab9beca62f7228182f6bc87c946572"}
Jul 10 19:20:48.867610 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.867502    2561 scope.go:117] "RemoveContainer" containerID="dabbd3fc70aab1484087b8268f3e6b1d7721ad277316a02978f189cd1cd2d34b"
Jul 10 19:20:48.877000 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.876983    2561 scope.go:117] "RemoveContainer" containerID="eef29f1a007cf571fcc181bbe4c317e97468a535b20ab023c2a532288418547b"
Jul 10 19:20:48.885594 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.885575    2561 scope.go:117] "RemoveContainer" containerID="6449a199825208a68733957aab5c311b076cfcd6dd31830fa98392f571bad613"
Jul 10 19:20:48.889584 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.889559    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"]
Jul 10 19:20:48.894050 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.894025    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc"]
Jul 10 19:20:48.894515 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.894499    2561 scope.go:117] "RemoveContainer" containerID="dabbd3fc70aab1484087b8268f3e6b1d7721ad277316a02978f189cd1cd2d34b"
Jul 10 19:20:48.894832 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:20:48.894813    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dabbd3fc70aab1484087b8268f3e6b1d7721ad277316a02978f189cd1cd2d34b\": container with ID starting with dabbd3fc70aab1484087b8268f3e6b1d7721ad277316a02978f189cd1cd2d34b not found: ID does not exist" containerID="dabbd3fc70aab1484087b8268f3e6b1d7721ad277316a02978f189cd1cd2d34b"
Jul 10 19:20:48.894895 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.894840    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dabbd3fc70aab1484087b8268f3e6b1d7721ad277316a02978f189cd1cd2d34b"} err="failed to get container status \"dabbd3fc70aab1484087b8268f3e6b1d7721ad277316a02978f189cd1cd2d34b\": rpc error: code = NotFound desc = could not find container \"dabbd3fc70aab1484087b8268f3e6b1d7721ad277316a02978f189cd1cd2d34b\": container with ID starting with dabbd3fc70aab1484087b8268f3e6b1d7721ad277316a02978f189cd1cd2d34b not found: ID does not exist"
Jul 10 19:20:48.894895 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.894860    2561 scope.go:117] "RemoveContainer" containerID="eef29f1a007cf571fcc181bbe4c317e97468a535b20ab023c2a532288418547b"
Jul 10 19:20:48.895129 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:20:48.895107    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eef29f1a007cf571fcc181bbe4c317e97468a535b20ab023c2a532288418547b\": container with ID starting with eef29f1a007cf571fcc181bbe4c317e97468a535b20ab023c2a532288418547b not found: ID does not exist" containerID="eef29f1a007cf571fcc181bbe4c317e97468a535b20ab023c2a532288418547b"
Jul 10 19:20:48.895175 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.895136    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef29f1a007cf571fcc181bbe4c317e97468a535b20ab023c2a532288418547b"} err="failed to get container status \"eef29f1a007cf571fcc181bbe4c317e97468a535b20ab023c2a532288418547b\": rpc error: code = NotFound desc = could not find container \"eef29f1a007cf571fcc181bbe4c317e97468a535b20ab023c2a532288418547b\": container with ID starting with eef29f1a007cf571fcc181bbe4c317e97468a535b20ab023c2a532288418547b not found: ID does not exist"
Jul 10 19:20:48.895175 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.895155    2561 scope.go:117] "RemoveContainer" containerID="6449a199825208a68733957aab5c311b076cfcd6dd31830fa98392f571bad613"
Jul 10 19:20:48.895400 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:20:48.895381    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6449a199825208a68733957aab5c311b076cfcd6dd31830fa98392f571bad613\": container with ID starting with 6449a199825208a68733957aab5c311b076cfcd6dd31830fa98392f571bad613 not found: ID does not exist" containerID="6449a199825208a68733957aab5c311b076cfcd6dd31830fa98392f571bad613"
Jul 10 19:20:48.895450 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:48.895409    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6449a199825208a68733957aab5c311b076cfcd6dd31830fa98392f571bad613"} err="failed to get container status \"6449a199825208a68733957aab5c311b076cfcd6dd31830fa98392f571bad613\": rpc error: code = NotFound desc = could not find container \"6449a199825208a68733957aab5c311b076cfcd6dd31830fa98392f571bad613\": container with ID starting with 6449a199825208a68733957aab5c311b076cfcd6dd31830fa98392f571bad613 not found: ID does not exist"
Jul 10 19:20:49.072342 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:49.072309    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c01643a-50e4-4e66-b131-47e6565ec206" path="/var/lib/kubelet/pods/8c01643a-50e4-4e66-b131-47e6565ec206/volumes"
Jul 10 19:20:55.267302 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.267257    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:20:55.459944 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.459903    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"]
Jul 10 19:20:55.460329 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.460315    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c01643a-50e4-4e66-b131-47e6565ec206" containerName="tokenizer"
Jul 10 19:20:55.460329 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.460330    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c01643a-50e4-4e66-b131-47e6565ec206" containerName="tokenizer"
Jul 10 19:20:55.460415 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.460348    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c01643a-50e4-4e66-b131-47e6565ec206" containerName="storage-initializer"
Jul 10 19:20:55.460415 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.460354    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c01643a-50e4-4e66-b131-47e6565ec206" containerName="storage-initializer"
Jul 10 19:20:55.460415 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.460366    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c01643a-50e4-4e66-b131-47e6565ec206" containerName="main"
Jul 10 19:20:55.460415 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.460372    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c01643a-50e4-4e66-b131-47e6565ec206" containerName="main"
Jul 10 19:20:55.460537 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.460431    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c01643a-50e4-4e66-b131-47e6565ec206" containerName="main"
Jul 10 19:20:55.460537 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.460440    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c01643a-50e4-4e66-b131-47e6565ec206" containerName="tokenizer"
Jul 10 19:20:55.465357 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.465337    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.468179 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.468153    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 10 19:20:55.468347 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.468156    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-k4z66\""
Jul 10 19:20:55.478855 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.478827    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"]
Jul 10 19:20:55.514238 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.514203    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:20:55.601788 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.601705    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.601788 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.601787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.601983 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.601807    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.601983 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.601833    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmfcq\" (UniqueName: \"kubernetes.io/projected/2a40a106-37e5-4d2b-bf47-c671f5fb6808-kube-api-access-tmfcq\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.601983 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.601854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.601983 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.601907    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.702493 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.702457    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.702493 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.702496    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.702710 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.702624    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmfcq\" (UniqueName: \"kubernetes.io/projected/2a40a106-37e5-4d2b-bf47-c671f5fb6808-kube-api-access-tmfcq\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.702710 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.702661    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.702806 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.702709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.702806 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.702755    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.702951 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.702928    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.703018 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.702968    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.703108 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.703050    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.703195 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.703175    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.705047 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.705024    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.710801 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.710770    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmfcq\" (UniqueName: \"kubernetes.io/projected/2a40a106-37e5-4d2b-bf47-c671f5fb6808-kube-api-access-tmfcq\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.776472 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.776421    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:55.927842 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:55.927814    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"]
Jul 10 19:20:55.930433 ip-10-0-135-91 kubenswrapper[2561]: W0710 19:20:55.930405    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a40a106_37e5_4d2b_bf47_c671f5fb6808.slice/crio-5d8581ad5e6846f3eb55cd554be2b07210d4f899f2e0b630d3292a8f3e5de890 WatchSource:0}: Error finding container 5d8581ad5e6846f3eb55cd554be2b07210d4f899f2e0b630d3292a8f3e5de890: Status 404 returned error can't find the container with id 5d8581ad5e6846f3eb55cd554be2b07210d4f899f2e0b630d3292a8f3e5de890
Jul 10 19:20:56.910504 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:56.910468    2561 generic.go:356] "Generic (PLEG): container finished" podID="2a40a106-37e5-4d2b-bf47-c671f5fb6808" containerID="dfc29755a335d6b45e074890afe55e225d316272778521a23ff4d3d306a327ed" exitCode=0
Jul 10 19:20:56.910888 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:56.910567    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8" event={"ID":"2a40a106-37e5-4d2b-bf47-c671f5fb6808","Type":"ContainerDied","Data":"dfc29755a335d6b45e074890afe55e225d316272778521a23ff4d3d306a327ed"}
Jul 10 19:20:56.910888 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:56.910622    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8" event={"ID":"2a40a106-37e5-4d2b-bf47-c671f5fb6808","Type":"ContainerStarted","Data":"5d8581ad5e6846f3eb55cd554be2b07210d4f899f2e0b630d3292a8f3e5de890"}
Jul 10 19:20:57.918046 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:57.918003    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8" event={"ID":"2a40a106-37e5-4d2b-bf47-c671f5fb6808","Type":"ContainerStarted","Data":"3a0023c6747be9492b8fe74bb7360da5405f40b36451b8545541daca3c4b122d"}
Jul 10 19:20:57.918497 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:57.918068    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8" event={"ID":"2a40a106-37e5-4d2b-bf47-c671f5fb6808","Type":"ContainerStarted","Data":"9c7f2aae6153a14279c7aed1c0feef00bf6da57d9fdc0d046fe15b4dcc9aa7cd"}
Jul 10 19:20:57.918497 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:57.918235    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:20:57.941127 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:20:57.941059    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8" podStartSLOduration=2.247548333 podStartE2EDuration="2.941040198s" totalImagesPullingTime="0s" totalInitContainerRuntime="693.491865ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:20:57.939161076 +0000 UTC m=+2481.413253973" watchObservedRunningTime="2026-07-10 19:20:57.941040198 +0000 UTC m=+2481.415133084"
Jul 10 19:21:05.777326 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:05.777286    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:21:05.777826 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:05.777361    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:21:05.780247 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:05.780213    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:21:05.960218 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:05.960178    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:21:09.157178 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:09.157144    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:21:22.041404 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:22.041368    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:21:22.082889 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:22.082856    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"]
Jul 10 19:21:22.083146 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:22.083116    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9" podUID="f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" containerName="main" containerID="cri-o://8b8026370fc3e193f39d9cdaa6559fcf949165b2770127366ea81c0bbd2136f8" gracePeriod=30
Jul 10 19:21:22.083306 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:22.083203    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9" podUID="f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" containerName="tokenizer" containerID="cri-o://22cf9008afd01f71cec2875b6ba6f65cd8f973fdfae31256d4317914ebd6d85a" gracePeriod=30
Jul 10 19:21:22.141362 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:22.141327    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:21:22.209911 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:22.209882    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:21:23.045356 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.045320    2561 generic.go:356] "Generic (PLEG): container finished" podID="f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" containerID="8b8026370fc3e193f39d9cdaa6559fcf949165b2770127366ea81c0bbd2136f8" exitCode=0
Jul 10 19:21:23.045731 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.045394    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9" event={"ID":"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd","Type":"ContainerDied","Data":"8b8026370fc3e193f39d9cdaa6559fcf949165b2770127366ea81c0bbd2136f8"}
Jul 10 19:21:23.448267 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.448239    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:21:23.455001 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.454980    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tls-certs\") pod \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") "
Jul 10 19:21:23.455110 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.455031    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tokenizer-cache\") pod \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") "
Jul 10 19:21:23.455110 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.455050    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tokenizer-uds\") pod \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") "
Jul 10 19:21:23.455110 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.455092    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c8nt\" (UniqueName: \"kubernetes.io/projected/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-kube-api-access-4c8nt\") pod \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") "
Jul 10 19:21:23.455252 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.455136    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tokenizer-tmp\") pod \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") "
Jul 10 19:21:23.455252 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.455200    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-kserve-provision-location\") pod \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\" (UID: \"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd\") "
Jul 10 19:21:23.455333 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.455263    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" (UID: "f8b3c1ee-3308-4474-a844-a94e1b8d0dcd"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:23.455394 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.455347    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" (UID: "f8b3c1ee-3308-4474-a844-a94e1b8d0dcd"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:23.455485 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.455466    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tokenizer-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:23.455485 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.455484    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tokenizer-uds\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:23.455637 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.455575    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" (UID: "f8b3c1ee-3308-4474-a844-a94e1b8d0dcd"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:23.455979 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.455956    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" (UID: "f8b3c1ee-3308-4474-a844-a94e1b8d0dcd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:23.457379 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.457355    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" (UID: "f8b3c1ee-3308-4474-a844-a94e1b8d0dcd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:21:23.457483 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.457411    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-kube-api-access-4c8nt" (OuterVolumeSpecName: "kube-api-access-4c8nt") pod "f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" (UID: "f8b3c1ee-3308-4474-a844-a94e1b8d0dcd"). InnerVolumeSpecName "kube-api-access-4c8nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:21:23.556134 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.556028    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-kserve-provision-location\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:23.556134 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.556060    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:23.556134 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.556098    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c8nt\" (UniqueName: \"kubernetes.io/projected/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-kube-api-access-4c8nt\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:23.556134 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:23.556111    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd-tokenizer-tmp\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:24.052093 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:24.052044    2561 generic.go:356] "Generic (PLEG): container finished" podID="f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" containerID="22cf9008afd01f71cec2875b6ba6f65cd8f973fdfae31256d4317914ebd6d85a" exitCode=0
Jul 10 19:21:24.052580 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:24.052119    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"
Jul 10 19:21:24.052580 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:24.052117    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9" event={"ID":"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd","Type":"ContainerDied","Data":"22cf9008afd01f71cec2875b6ba6f65cd8f973fdfae31256d4317914ebd6d85a"}
Jul 10 19:21:24.052580 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:24.052255    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9" event={"ID":"f8b3c1ee-3308-4474-a844-a94e1b8d0dcd","Type":"ContainerDied","Data":"ce2c3fb8e6939ca650127fee6cd6cb304eabc439d0cd40f2c8c96424af6d452c"}
Jul 10 19:21:24.052580 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:24.052281    2561 scope.go:117] "RemoveContainer" containerID="22cf9008afd01f71cec2875b6ba6f65cd8f973fdfae31256d4317914ebd6d85a"
Jul 10 19:21:24.061144 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:24.061123    2561 scope.go:117] "RemoveContainer" containerID="8b8026370fc3e193f39d9cdaa6559fcf949165b2770127366ea81c0bbd2136f8"
Jul 10 19:21:24.069188 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:24.069169    2561 scope.go:117] "RemoveContainer" containerID="88ed31d3e939b7f9f7931d2be3ed9bdc0dc4d15b6d802b0c371c0ca7e8788e17"
Jul 10 19:21:24.078014 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:24.077942    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"]
Jul 10 19:21:24.078176 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:24.078027    2561 scope.go:117] "RemoveContainer" containerID="22cf9008afd01f71cec2875b6ba6f65cd8f973fdfae31256d4317914ebd6d85a"
Jul 10 19:21:24.078407 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:21:24.078388    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22cf9008afd01f71cec2875b6ba6f65cd8f973fdfae31256d4317914ebd6d85a\": container with ID starting with 22cf9008afd01f71cec2875b6ba6f65cd8f973fdfae31256d4317914ebd6d85a not found: ID does not exist" containerID="22cf9008afd01f71cec2875b6ba6f65cd8f973fdfae31256d4317914ebd6d85a"
Jul 10 19:21:24.078454 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:24.078418    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22cf9008afd01f71cec2875b6ba6f65cd8f973fdfae31256d4317914ebd6d85a"} err="failed to get container status \"22cf9008afd01f71cec2875b6ba6f65cd8f973fdfae31256d4317914ebd6d85a\": rpc error: code = NotFound desc = could not find container \"22cf9008afd01f71cec2875b6ba6f65cd8f973fdfae31256d4317914ebd6d85a\": container with ID starting with 22cf9008afd01f71cec2875b6ba6f65cd8f973fdfae31256d4317914ebd6d85a not found: ID does not exist"
Jul 10 19:21:24.078454 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:24.078437    2561 scope.go:117] "RemoveContainer" containerID="8b8026370fc3e193f39d9cdaa6559fcf949165b2770127366ea81c0bbd2136f8"
Jul 10 19:21:24.078716 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:21:24.078692    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b8026370fc3e193f39d9cdaa6559fcf949165b2770127366ea81c0bbd2136f8\": container with ID starting with 8b8026370fc3e193f39d9cdaa6559fcf949165b2770127366ea81c0bbd2136f8 not found: ID does not exist" containerID="8b8026370fc3e193f39d9cdaa6559fcf949165b2770127366ea81c0bbd2136f8"
Jul 10 19:21:24.078760 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:24.078728    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b8026370fc3e193f39d9cdaa6559fcf949165b2770127366ea81c0bbd2136f8"} err="failed to get container status \"8b8026370fc3e193f39d9cdaa6559fcf949165b2770127366ea81c0bbd2136f8\": rpc error: code = NotFound desc = could not find container \"8b8026370fc3e193f39d9cdaa6559fcf949165b2770127366ea81c0bbd2136f8\": container with ID starting with 8b8026370fc3e193f39d9cdaa6559fcf949165b2770127366ea81c0bbd2136f8 not found: ID does not exist"
Jul 10 19:21:24.078760 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:24.078754    2561 scope.go:117] "RemoveContainer" containerID="88ed31d3e939b7f9f7931d2be3ed9bdc0dc4d15b6d802b0c371c0ca7e8788e17"
Jul 10 19:21:24.078988 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:21:24.078970    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ed31d3e939b7f9f7931d2be3ed9bdc0dc4d15b6d802b0c371c0ca7e8788e17\": container with ID starting with 88ed31d3e939b7f9f7931d2be3ed9bdc0dc4d15b6d802b0c371c0ca7e8788e17 not found: ID does not exist" containerID="88ed31d3e939b7f9f7931d2be3ed9bdc0dc4d15b6d802b0c371c0ca7e8788e17"
Jul 10 19:21:24.079037 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:24.078996    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ed31d3e939b7f9f7931d2be3ed9bdc0dc4d15b6d802b0c371c0ca7e8788e17"} err="failed to get container status \"88ed31d3e939b7f9f7931d2be3ed9bdc0dc4d15b6d802b0c371c0ca7e8788e17\": rpc error: code = NotFound desc = could not find container \"88ed31d3e939b7f9f7931d2be3ed9bdc0dc4d15b6d802b0c371c0ca7e8788e17\": container with ID starting with 88ed31d3e939b7f9f7931d2be3ed9bdc0dc4d15b6d802b0c371c0ca7e8788e17 not found: ID does not exist"
Jul 10 19:21:24.082829 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:24.082807    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9"]
Jul 10 19:21:25.072195 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:25.072156    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" path="/var/lib/kubelet/pods/f8b3c1ee-3308-4474-a844-a94e1b8d0dcd/volumes"
Jul 10 19:21:26.966515 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:21:26.966485    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:22:51.184590 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:22:51.184545    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:22:53.171536 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:22:53.171503    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:23:27.313888 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:27.313843    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:23:27.324705 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:27.324671    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"]
Jul 10 19:23:27.325007 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:27.324957    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8" podUID="2a40a106-37e5-4d2b-bf47-c671f5fb6808" containerName="main" containerID="cri-o://9c7f2aae6153a14279c7aed1c0feef00bf6da57d9fdc0d046fe15b4dcc9aa7cd" gracePeriod=30
Jul 10 19:23:27.325007 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:27.324987    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8" podUID="2a40a106-37e5-4d2b-bf47-c671f5fb6808" containerName="tokenizer" containerID="cri-o://3a0023c6747be9492b8fe74bb7360da5405f40b36451b8545541daca3c4b122d" gracePeriod=30
Jul 10 19:23:27.372128 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:27.372066    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:23:27.637952 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:27.637854    2561 generic.go:356] "Generic (PLEG): container finished" podID="2a40a106-37e5-4d2b-bf47-c671f5fb6808" containerID="9c7f2aae6153a14279c7aed1c0feef00bf6da57d9fdc0d046fe15b4dcc9aa7cd" exitCode=0
Jul 10 19:23:27.637952 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:27.637941    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8" event={"ID":"2a40a106-37e5-4d2b-bf47-c671f5fb6808","Type":"ContainerDied","Data":"9c7f2aae6153a14279c7aed1c0feef00bf6da57d9fdc0d046fe15b4dcc9aa7cd"}
Jul 10 19:23:28.592515 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.592488    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:23:28.644965 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.644934    2561 generic.go:356] "Generic (PLEG): container finished" podID="2a40a106-37e5-4d2b-bf47-c671f5fb6808" containerID="3a0023c6747be9492b8fe74bb7360da5405f40b36451b8545541daca3c4b122d" exitCode=0
Jul 10 19:23:28.645169 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.644990    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"
Jul 10 19:23:28.645169 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.645021    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8" event={"ID":"2a40a106-37e5-4d2b-bf47-c671f5fb6808","Type":"ContainerDied","Data":"3a0023c6747be9492b8fe74bb7360da5405f40b36451b8545541daca3c4b122d"}
Jul 10 19:23:28.645169 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.645113    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8" event={"ID":"2a40a106-37e5-4d2b-bf47-c671f5fb6808","Type":"ContainerDied","Data":"5d8581ad5e6846f3eb55cd554be2b07210d4f899f2e0b630d3292a8f3e5de890"}
Jul 10 19:23:28.645169 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.645150    2561 scope.go:117] "RemoveContainer" containerID="3a0023c6747be9492b8fe74bb7360da5405f40b36451b8545541daca3c4b122d"
Jul 10 19:23:28.653376 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.653359    2561 scope.go:117] "RemoveContainer" containerID="9c7f2aae6153a14279c7aed1c0feef00bf6da57d9fdc0d046fe15b4dcc9aa7cd"
Jul 10 19:23:28.661307 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.661286    2561 scope.go:117] "RemoveContainer" containerID="dfc29755a335d6b45e074890afe55e225d316272778521a23ff4d3d306a327ed"
Jul 10 19:23:28.669397 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.669376    2561 scope.go:117] "RemoveContainer" containerID="3a0023c6747be9492b8fe74bb7360da5405f40b36451b8545541daca3c4b122d"
Jul 10 19:23:28.669675 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:23:28.669649    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0023c6747be9492b8fe74bb7360da5405f40b36451b8545541daca3c4b122d\": container with ID starting with 3a0023c6747be9492b8fe74bb7360da5405f40b36451b8545541daca3c4b122d not found: ID does not exist" containerID="3a0023c6747be9492b8fe74bb7360da5405f40b36451b8545541daca3c4b122d"
Jul 10 19:23:28.669739 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.669690    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0023c6747be9492b8fe74bb7360da5405f40b36451b8545541daca3c4b122d"} err="failed to get container status \"3a0023c6747be9492b8fe74bb7360da5405f40b36451b8545541daca3c4b122d\": rpc error: code = NotFound desc = could not find container \"3a0023c6747be9492b8fe74bb7360da5405f40b36451b8545541daca3c4b122d\": container with ID starting with 3a0023c6747be9492b8fe74bb7360da5405f40b36451b8545541daca3c4b122d not found: ID does not exist"
Jul 10 19:23:28.669739 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.669710    2561 scope.go:117] "RemoveContainer" containerID="9c7f2aae6153a14279c7aed1c0feef00bf6da57d9fdc0d046fe15b4dcc9aa7cd"
Jul 10 19:23:28.669970 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:23:28.669951    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c7f2aae6153a14279c7aed1c0feef00bf6da57d9fdc0d046fe15b4dcc9aa7cd\": container with ID starting with 9c7f2aae6153a14279c7aed1c0feef00bf6da57d9fdc0d046fe15b4dcc9aa7cd not found: ID does not exist" containerID="9c7f2aae6153a14279c7aed1c0feef00bf6da57d9fdc0d046fe15b4dcc9aa7cd"
Jul 10 19:23:28.670042 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.669980    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c7f2aae6153a14279c7aed1c0feef00bf6da57d9fdc0d046fe15b4dcc9aa7cd"} err="failed to get container status \"9c7f2aae6153a14279c7aed1c0feef00bf6da57d9fdc0d046fe15b4dcc9aa7cd\": rpc error: code = NotFound desc = could not find container \"9c7f2aae6153a14279c7aed1c0feef00bf6da57d9fdc0d046fe15b4dcc9aa7cd\": container with ID starting with 9c7f2aae6153a14279c7aed1c0feef00bf6da57d9fdc0d046fe15b4dcc9aa7cd not found: ID does not exist"
Jul 10 19:23:28.670042 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.669997    2561 scope.go:117] "RemoveContainer" containerID="dfc29755a335d6b45e074890afe55e225d316272778521a23ff4d3d306a327ed"
Jul 10 19:23:28.670228 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:23:28.670209    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfc29755a335d6b45e074890afe55e225d316272778521a23ff4d3d306a327ed\": container with ID starting with dfc29755a335d6b45e074890afe55e225d316272778521a23ff4d3d306a327ed not found: ID does not exist" containerID="dfc29755a335d6b45e074890afe55e225d316272778521a23ff4d3d306a327ed"
Jul 10 19:23:28.670284 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.670237    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfc29755a335d6b45e074890afe55e225d316272778521a23ff4d3d306a327ed"} err="failed to get container status \"dfc29755a335d6b45e074890afe55e225d316272778521a23ff4d3d306a327ed\": rpc error: code = NotFound desc = could not find container \"dfc29755a335d6b45e074890afe55e225d316272778521a23ff4d3d306a327ed\": container with ID starting with dfc29755a335d6b45e074890afe55e225d316272778521a23ff4d3d306a327ed not found: ID does not exist"
Jul 10 19:23:28.679513 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.679495    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-kserve-provision-location\") pod \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") "
Jul 10 19:23:28.679580 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.679523    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tokenizer-cache\") pod \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") "
Jul 10 19:23:28.679580 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.679573    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tokenizer-uds\") pod \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") "
Jul 10 19:23:28.679643 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.679609    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmfcq\" (UniqueName: \"kubernetes.io/projected/2a40a106-37e5-4d2b-bf47-c671f5fb6808-kube-api-access-tmfcq\") pod \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") "
Jul 10 19:23:28.679676 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.679651    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tls-certs\") pod \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") "
Jul 10 19:23:28.679711 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.679688    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tokenizer-tmp\") pod \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\" (UID: \"2a40a106-37e5-4d2b-bf47-c671f5fb6808\") "
Jul 10 19:23:28.679854 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.679832    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2a40a106-37e5-4d2b-bf47-c671f5fb6808" (UID: "2a40a106-37e5-4d2b-bf47-c671f5fb6808"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:23:28.679928 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.679849    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "2a40a106-37e5-4d2b-bf47-c671f5fb6808" (UID: "2a40a106-37e5-4d2b-bf47-c671f5fb6808"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:23:28.679983 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.679952    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tokenizer-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:23:28.679983 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.679970    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tokenizer-uds\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:23:28.680140 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.680116    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "2a40a106-37e5-4d2b-bf47-c671f5fb6808" (UID: "2a40a106-37e5-4d2b-bf47-c671f5fb6808"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:23:28.680396 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.680377    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a40a106-37e5-4d2b-bf47-c671f5fb6808" (UID: "2a40a106-37e5-4d2b-bf47-c671f5fb6808"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:23:28.681703 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.681667    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a40a106-37e5-4d2b-bf47-c671f5fb6808" (UID: "2a40a106-37e5-4d2b-bf47-c671f5fb6808"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:23:28.681799 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.681744    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a40a106-37e5-4d2b-bf47-c671f5fb6808-kube-api-access-tmfcq" (OuterVolumeSpecName: "kube-api-access-tmfcq") pod "2a40a106-37e5-4d2b-bf47-c671f5fb6808" (UID: "2a40a106-37e5-4d2b-bf47-c671f5fb6808"). InnerVolumeSpecName "kube-api-access-tmfcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:23:28.780805 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.780769    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmfcq\" (UniqueName: \"kubernetes.io/projected/2a40a106-37e5-4d2b-bf47-c671f5fb6808-kube-api-access-tmfcq\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:23:28.780805 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.780798    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:23:28.780805 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.780809    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-tokenizer-tmp\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:23:28.781055 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.780818    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a40a106-37e5-4d2b-bf47-c671f5fb6808-kserve-provision-location\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:23:28.809803 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.809767    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dmsgq/must-gather-xpqqb"]
Jul 10 19:23:28.810143 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.810131    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" containerName="main"
Jul 10 19:23:28.810193 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.810145    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" containerName="main"
Jul 10 19:23:28.810193 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.810154    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a40a106-37e5-4d2b-bf47-c671f5fb6808" containerName="tokenizer"
Jul 10 19:23:28.810193 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.810159    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a40a106-37e5-4d2b-bf47-c671f5fb6808" containerName="tokenizer"
Jul 10 19:23:28.810193 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.810168    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" containerName="tokenizer"
Jul 10 19:23:28.810193 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.810173    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" containerName="tokenizer"
Jul 10 19:23:28.810193 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.810181    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a40a106-37e5-4d2b-bf47-c671f5fb6808" containerName="storage-initializer"
Jul 10 19:23:28.810193 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.810186    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a40a106-37e5-4d2b-bf47-c671f5fb6808" containerName="storage-initializer"
Jul 10 19:23:28.810193 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.810193    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a40a106-37e5-4d2b-bf47-c671f5fb6808" containerName="main"
Jul 10 19:23:28.810428 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.810198    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a40a106-37e5-4d2b-bf47-c671f5fb6808" containerName="main"
Jul 10 19:23:28.810428 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.810209    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" containerName="storage-initializer"
Jul 10 19:23:28.810428 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.810215    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" containerName="storage-initializer"
Jul 10 19:23:28.810428 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.810275    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" containerName="main"
Jul 10 19:23:28.810428 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.810283    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a40a106-37e5-4d2b-bf47-c671f5fb6808" containerName="tokenizer"
Jul 10 19:23:28.810428 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.810290    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8b3c1ee-3308-4474-a844-a94e1b8d0dcd" containerName="tokenizer"
Jul 10 19:23:28.810428 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.810298    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a40a106-37e5-4d2b-bf47-c671f5fb6808" containerName="main"
Jul 10 19:23:28.814952 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.814932    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dmsgq/must-gather-xpqqb"
Jul 10 19:23:28.817295 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.817274    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dmsgq\"/\"kube-root-ca.crt\""
Jul 10 19:23:28.817400 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.817277    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dmsgq\"/\"default-dockercfg-9hnzw\""
Jul 10 19:23:28.817400 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.817338    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dmsgq\"/\"openshift-service-ca.crt\""
Jul 10 19:23:28.823071 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.823049    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dmsgq/must-gather-xpqqb"]
Jul 10 19:23:28.827302 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.827275    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:23:28.894133 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.894096    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:23:28.966873 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.966830    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"]
Jul 10 19:23:28.973014 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.972986    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8"]
Jul 10 19:23:28.982792 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.982769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/78501dc8-ebd7-44b4-8c24-fefc3fd67900-must-gather-output\") pod \"must-gather-xpqqb\" (UID: \"78501dc8-ebd7-44b4-8c24-fefc3fd67900\") " pod="openshift-must-gather-dmsgq/must-gather-xpqqb"
Jul 10 19:23:28.982882 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:28.982806    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqshr\" (UniqueName: \"kubernetes.io/projected/78501dc8-ebd7-44b4-8c24-fefc3fd67900-kube-api-access-fqshr\") pod \"must-gather-xpqqb\" (UID: \"78501dc8-ebd7-44b4-8c24-fefc3fd67900\") " pod="openshift-must-gather-dmsgq/must-gather-xpqqb"
Jul 10 19:23:29.074051 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:29.074014    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a40a106-37e5-4d2b-bf47-c671f5fb6808" path="/var/lib/kubelet/pods/2a40a106-37e5-4d2b-bf47-c671f5fb6808/volumes"
Jul 10 19:23:29.084178 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:29.084155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/78501dc8-ebd7-44b4-8c24-fefc3fd67900-must-gather-output\") pod \"must-gather-xpqqb\" (UID: \"78501dc8-ebd7-44b4-8c24-fefc3fd67900\") " pod="openshift-must-gather-dmsgq/must-gather-xpqqb"
Jul 10 19:23:29.084307 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:29.084190    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqshr\" (UniqueName: \"kubernetes.io/projected/78501dc8-ebd7-44b4-8c24-fefc3fd67900-kube-api-access-fqshr\") pod \"must-gather-xpqqb\" (UID: \"78501dc8-ebd7-44b4-8c24-fefc3fd67900\") " pod="openshift-must-gather-dmsgq/must-gather-xpqqb"
Jul 10 19:23:29.084494 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:29.084475    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/78501dc8-ebd7-44b4-8c24-fefc3fd67900-must-gather-output\") pod \"must-gather-xpqqb\" (UID: \"78501dc8-ebd7-44b4-8c24-fefc3fd67900\") " pod="openshift-must-gather-dmsgq/must-gather-xpqqb"
Jul 10 19:23:29.091899 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:29.091879    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqshr\" (UniqueName: \"kubernetes.io/projected/78501dc8-ebd7-44b4-8c24-fefc3fd67900-kube-api-access-fqshr\") pod \"must-gather-xpqqb\" (UID: \"78501dc8-ebd7-44b4-8c24-fefc3fd67900\") " pod="openshift-must-gather-dmsgq/must-gather-xpqqb"
Jul 10 19:23:29.124743 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:29.124705    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dmsgq/must-gather-xpqqb"
Jul 10 19:23:29.257154 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:29.257129    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dmsgq/must-gather-xpqqb"]
Jul 10 19:23:29.259789 ip-10-0-135-91 kubenswrapper[2561]: W0710 19:23:29.259767    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78501dc8_ebd7_44b4_8c24_fefc3fd67900.slice/crio-0479c5563da6f5671ad49b36a41bf2d5a6ed5c82df84492bb3b81c986b6793cf WatchSource:0}: Error finding container 0479c5563da6f5671ad49b36a41bf2d5a6ed5c82df84492bb3b81c986b6793cf: Status 404 returned error can't find the container with id 0479c5563da6f5671ad49b36a41bf2d5a6ed5c82df84492bb3b81c986b6793cf
Jul 10 19:23:29.261493 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:29.261477    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 19:23:29.651140 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:29.651011    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dmsgq/must-gather-xpqqb" event={"ID":"78501dc8-ebd7-44b4-8c24-fefc3fd67900","Type":"ContainerStarted","Data":"0479c5563da6f5671ad49b36a41bf2d5a6ed5c82df84492bb3b81c986b6793cf"}
Jul 10 19:23:35.694972 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:35.694931    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dmsgq/must-gather-xpqqb" event={"ID":"78501dc8-ebd7-44b4-8c24-fefc3fd67900","Type":"ContainerStarted","Data":"b40214972e7df18e531bb704f359b8ea6da7210f9fe3dc8c8e4e4d3511f334a7"}
Jul 10 19:23:35.695427 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:35.694985    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dmsgq/must-gather-xpqqb" event={"ID":"78501dc8-ebd7-44b4-8c24-fefc3fd67900","Type":"ContainerStarted","Data":"84303cf8293b68ebbcb77999f4006802433389458f5555b8e713fe6a2b85c862"}
Jul 10 19:23:35.714283 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:23:35.714230    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dmsgq/must-gather-xpqqb" podStartSLOduration=1.572279733 podStartE2EDuration="7.714214116s" totalImagesPullingTime="6.141934383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:23:35.712658739 +0000 UTC m=+2639.186751623" watchObservedRunningTime="2026-07-10 19:23:35.714214116 +0000 UTC m=+2639.188306992"
Jul 10 19:24:02.598132 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:02.598099    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-c9n2w_c6ad8822-d4e5-4d82-843d-5f97b6847eb6/kuadrant-console-plugin/0.log"
Jul 10 19:24:02.787700 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:02.787671    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-tghfc_b8760a43-d559-4b40-9a46-161389401834/manager/0.log"
Jul 10 19:24:03.843730 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:03.843643    2561 generic.go:356] "Generic (PLEG): container finished" podID="78501dc8-ebd7-44b4-8c24-fefc3fd67900" containerID="84303cf8293b68ebbcb77999f4006802433389458f5555b8e713fe6a2b85c862" exitCode=0
Jul 10 19:24:03.843730 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:03.843691    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dmsgq/must-gather-xpqqb" event={"ID":"78501dc8-ebd7-44b4-8c24-fefc3fd67900","Type":"ContainerDied","Data":"84303cf8293b68ebbcb77999f4006802433389458f5555b8e713fe6a2b85c862"}
Jul 10 19:24:03.844222 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:03.843967    2561 scope.go:117] "RemoveContainer" containerID="84303cf8293b68ebbcb77999f4006802433389458f5555b8e713fe6a2b85c862"
Jul 10 19:24:04.485058 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:04.485029    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-dmsgq_must-gather-xpqqb_78501dc8-ebd7-44b4-8c24-fefc3fd67900/gather/0.log"
Jul 10 19:24:04.993487 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:04.993456    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:24:05.161524 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:05.161489    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:24:05.229764 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:05.229729    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:24:08.235047 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:08.235014    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nz2c7_cde9c141-cf6b-44ff-b24d-15e1afda3808/global-pull-secret-syncer/0.log"
Jul 10 19:24:08.305794 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:08.305764    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log"
Jul 10 19:24:08.430063 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:08.430024    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-91.ec2.internal_f9df3b169ce66d03811239efffa50458/haproxy/0.log"
Jul 10 19:24:09.970845 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:09.970804    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-dmsgq/must-gather-xpqqb"]
Jul 10 19:24:09.971309 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:09.970986    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-dmsgq/must-gather-xpqqb" podUID="78501dc8-ebd7-44b4-8c24-fefc3fd67900" containerName="copy" containerID="cri-o://b40214972e7df18e531bb704f359b8ea6da7210f9fe3dc8c8e4e4d3511f334a7" gracePeriod=2
Jul 10 19:24:09.973337 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:24:09.973296    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xpqqb\" is forbidden: User \"system:node:ip-10-0-135-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dmsgq\": no relationship found between node 'ip-10-0-135-91.ec2.internal' and this object" podUID="78501dc8-ebd7-44b4-8c24-fefc3fd67900" pod="openshift-must-gather-dmsgq/must-gather-xpqqb"
Jul 10 19:24:09.975314 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:09.975288    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-dmsgq/must-gather-xpqqb"]
Jul 10 19:24:10.212435 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:10.212411    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-dmsgq_must-gather-xpqqb_78501dc8-ebd7-44b4-8c24-fefc3fd67900/copy/0.log"
Jul 10 19:24:10.215161 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:10.215135    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dmsgq/must-gather-xpqqb"
Jul 10 19:24:10.217237 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:24:10.217214    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xpqqb\" is forbidden: User \"system:node:ip-10-0-135-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dmsgq\": no relationship found between node 'ip-10-0-135-91.ec2.internal' and this object" podUID="78501dc8-ebd7-44b4-8c24-fefc3fd67900" pod="openshift-must-gather-dmsgq/must-gather-xpqqb"
Jul 10 19:24:10.362902 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:10.362826    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/78501dc8-ebd7-44b4-8c24-fefc3fd67900-must-gather-output\") pod \"78501dc8-ebd7-44b4-8c24-fefc3fd67900\" (UID: \"78501dc8-ebd7-44b4-8c24-fefc3fd67900\") "
Jul 10 19:24:10.362902 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:10.362889    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqshr\" (UniqueName: \"kubernetes.io/projected/78501dc8-ebd7-44b4-8c24-fefc3fd67900-kube-api-access-fqshr\") pod \"78501dc8-ebd7-44b4-8c24-fefc3fd67900\" (UID: \"78501dc8-ebd7-44b4-8c24-fefc3fd67900\") "
Jul 10 19:24:10.365191 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:10.365155    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78501dc8-ebd7-44b4-8c24-fefc3fd67900-kube-api-access-fqshr" (OuterVolumeSpecName: "kube-api-access-fqshr") pod "78501dc8-ebd7-44b4-8c24-fefc3fd67900" (UID: "78501dc8-ebd7-44b4-8c24-fefc3fd67900"). InnerVolumeSpecName "kube-api-access-fqshr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:24:10.368955 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:10.368926    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78501dc8-ebd7-44b4-8c24-fefc3fd67900-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "78501dc8-ebd7-44b4-8c24-fefc3fd67900" (UID: "78501dc8-ebd7-44b4-8c24-fefc3fd67900"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:24:10.464141 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:10.464107    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqshr\" (UniqueName: \"kubernetes.io/projected/78501dc8-ebd7-44b4-8c24-fefc3fd67900-kube-api-access-fqshr\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:24:10.464141 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:10.464138    2561 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/78501dc8-ebd7-44b4-8c24-fefc3fd67900-must-gather-output\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 10 19:24:10.876271 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:10.876242    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-dmsgq_must-gather-xpqqb_78501dc8-ebd7-44b4-8c24-fefc3fd67900/copy/0.log"
Jul 10 19:24:10.876625 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:10.876599    2561 generic.go:356] "Generic (PLEG): container finished" podID="78501dc8-ebd7-44b4-8c24-fefc3fd67900" containerID="b40214972e7df18e531bb704f359b8ea6da7210f9fe3dc8c8e4e4d3511f334a7" exitCode=143
Jul 10 19:24:10.876706 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:10.876641    2561 scope.go:117] "RemoveContainer" containerID="b40214972e7df18e531bb704f359b8ea6da7210f9fe3dc8c8e4e4d3511f334a7"
Jul 10 19:24:10.876706 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:10.876648    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dmsgq/must-gather-xpqqb"
Jul 10 19:24:10.878983 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:24:10.878955    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xpqqb\" is forbidden: User \"system:node:ip-10-0-135-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dmsgq\": no relationship found between node 'ip-10-0-135-91.ec2.internal' and this object" podUID="78501dc8-ebd7-44b4-8c24-fefc3fd67900" pod="openshift-must-gather-dmsgq/must-gather-xpqqb"
Jul 10 19:24:10.886663 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:10.886639    2561 scope.go:117] "RemoveContainer" containerID="84303cf8293b68ebbcb77999f4006802433389458f5555b8e713fe6a2b85c862"
Jul 10 19:24:10.888996 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:24:10.888961    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xpqqb\" is forbidden: User \"system:node:ip-10-0-135-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dmsgq\": no relationship found between node 'ip-10-0-135-91.ec2.internal' and this object" podUID="78501dc8-ebd7-44b4-8c24-fefc3fd67900" pod="openshift-must-gather-dmsgq/must-gather-xpqqb"
Jul 10 19:24:10.900235 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:10.900201    2561 scope.go:117] "RemoveContainer" containerID="b40214972e7df18e531bb704f359b8ea6da7210f9fe3dc8c8e4e4d3511f334a7"
Jul 10 19:24:10.900552 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:24:10.900529    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b40214972e7df18e531bb704f359b8ea6da7210f9fe3dc8c8e4e4d3511f334a7\": container with ID starting with b40214972e7df18e531bb704f359b8ea6da7210f9fe3dc8c8e4e4d3511f334a7 not found: ID does not exist" containerID="b40214972e7df18e531bb704f359b8ea6da7210f9fe3dc8c8e4e4d3511f334a7"
Jul 10 19:24:10.900627 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:10.900561    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b40214972e7df18e531bb704f359b8ea6da7210f9fe3dc8c8e4e4d3511f334a7"} err="failed to get container status \"b40214972e7df18e531bb704f359b8ea6da7210f9fe3dc8c8e4e4d3511f334a7\": rpc error: code = NotFound desc = could not find container \"b40214972e7df18e531bb704f359b8ea6da7210f9fe3dc8c8e4e4d3511f334a7\": container with ID starting with b40214972e7df18e531bb704f359b8ea6da7210f9fe3dc8c8e4e4d3511f334a7 not found: ID does not exist"
Jul 10 19:24:10.900627 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:10.900578    2561 scope.go:117] "RemoveContainer" containerID="84303cf8293b68ebbcb77999f4006802433389458f5555b8e713fe6a2b85c862"
Jul 10 19:24:10.900824 ip-10-0-135-91 kubenswrapper[2561]: E0710 19:24:10.900808    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84303cf8293b68ebbcb77999f4006802433389458f5555b8e713fe6a2b85c862\": container with ID starting with 84303cf8293b68ebbcb77999f4006802433389458f5555b8e713fe6a2b85c862 not found: ID does not exist" containerID="84303cf8293b68ebbcb77999f4006802433389458f5555b8e713fe6a2b85c862"
Jul 10 19:24:10.900869 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:10.900827    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84303cf8293b68ebbcb77999f4006802433389458f5555b8e713fe6a2b85c862"} err="failed to get container status \"84303cf8293b68ebbcb77999f4006802433389458f5555b8e713fe6a2b85c862\": rpc error: code = NotFound desc = could not find container \"84303cf8293b68ebbcb77999f4006802433389458f5555b8e713fe6a2b85c862\": container with ID starting with 84303cf8293b68ebbcb77999f4006802433389458f5555b8e713fe6a2b85c862 not found: ID does not exist"
Jul 10 19:24:11.046263 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:11.046231    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:24:11.073370 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:11.073340    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78501dc8-ebd7-44b4-8c24-fefc3fd67900" path="/var/lib/kubelet/pods/78501dc8-ebd7-44b4-8c24-fefc3fd67900/volumes"
Jul 10 19:24:11.147357 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:11.147280    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6qx2_fe5d7751-950c-406a-8a69-bf619d6f98b6/konnectivity-agent/0.log" limit=1024
Jul 10 19:24:13.595922 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:13.595897    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-c9n2w_c6ad8822-d4e5-4d82-843d-5f97b6847eb6/kuadrant-console-plugin/0.log"
Jul 10 19:24:13.811563 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:13.811530    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-tghfc_b8760a43-d559-4b40-9a46-161389401834/manager/0.log"
Jul 10 19:24:14.932366 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:14.932336    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_66a2f044-d6d4-401c-b49e-19f73a1bb034/alertmanager/0.log"
Jul 10 19:24:14.959807 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:14.959776    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_66a2f044-d6d4-401c-b49e-19f73a1bb034/config-reloader/0.log"
Jul 10 19:24:14.998113 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:14.998070    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_66a2f044-d6d4-401c-b49e-19f73a1bb034/kube-rbac-proxy-web/0.log"
Jul 10 19:24:15.023823 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:15.023797    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_66a2f044-d6d4-401c-b49e-19f73a1bb034/kube-rbac-proxy/0.log"
Jul 10 19:24:15.048489 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:15.048461    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_66a2f044-d6d4-401c-b49e-19f73a1bb034/kube-rbac-proxy-metric/0.log"
Jul 10 19:24:15.074219 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:15.074186    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_66a2f044-d6d4-401c-b49e-19f73a1bb034/prom-label-proxy/0.log"
Jul 10 19:24:15.102991 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:15.102963    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_66a2f044-d6d4-401c-b49e-19f73a1bb034/init-config-reloader/0.log"
Jul 10 19:24:15.446541 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:15.446461    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f47gv_ebef81d2-c0d1-4894-9ca0-391f361b659b/node-exporter/0.log"
Jul 10 19:24:15.472664 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:15.472640    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f47gv_ebef81d2-c0d1-4894-9ca0-391f361b659b/kube-rbac-proxy/0.log"
Jul 10 19:24:15.501548 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:15.501525    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f47gv_ebef81d2-c0d1-4894-9ca0-391f361b659b/init-textfile/0.log"
Jul 10 19:24:15.628656 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:15.628621    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-d9kxp_1fb2c50d-7009-417b-927b-bbb8e45c94e0/kube-rbac-proxy-main/0.log"
Jul 10 19:24:15.652787 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:15.652760    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-d9kxp_1fb2c50d-7009-417b-927b-bbb8e45c94e0/kube-rbac-proxy-self/0.log"
Jul 10 19:24:15.680251 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:15.680216    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-d9kxp_1fb2c50d-7009-417b-927b-bbb8e45c94e0/openshift-state-metrics/0.log"
Jul 10 19:24:15.728440 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:15.728343    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a54a4c89-ddb8-4300-bab5-8fdc94b65977/prometheus/0.log"
Jul 10 19:24:15.750294 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:15.750272    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a54a4c89-ddb8-4300-bab5-8fdc94b65977/config-reloader/0.log"
Jul 10 19:24:15.780895 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:15.780869    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a54a4c89-ddb8-4300-bab5-8fdc94b65977/thanos-sidecar/0.log"
Jul 10 19:24:15.808341 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:15.808312    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a54a4c89-ddb8-4300-bab5-8fdc94b65977/kube-rbac-proxy-web/0.log"
Jul 10 19:24:15.833917 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:15.833891    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a54a4c89-ddb8-4300-bab5-8fdc94b65977/kube-rbac-proxy/0.log"
Jul 10 19:24:15.868442 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:15.868403    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a54a4c89-ddb8-4300-bab5-8fdc94b65977/kube-rbac-proxy-thanos/0.log"
Jul 10 19:24:15.905253 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:15.905213    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a54a4c89-ddb8-4300-bab5-8fdc94b65977/init-config-reloader/0.log"
Jul 10 19:24:16.122502 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:16.122472    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd7ccc499-69tw6_66182fd1-a634-4db8-b35e-d64b8632eadc/thanos-query/0.log"
Jul 10 19:24:16.154846 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:16.154815    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd7ccc499-69tw6_66182fd1-a634-4db8-b35e-d64b8632eadc/kube-rbac-proxy-web/0.log"
Jul 10 19:24:16.184582 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:16.184553    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd7ccc499-69tw6_66182fd1-a634-4db8-b35e-d64b8632eadc/kube-rbac-proxy/0.log"
Jul 10 19:24:16.207532 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:16.207510    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd7ccc499-69tw6_66182fd1-a634-4db8-b35e-d64b8632eadc/prom-label-proxy/0.log"
Jul 10 19:24:16.269537 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:16.269462    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd7ccc499-69tw6_66182fd1-a634-4db8-b35e-d64b8632eadc/kube-rbac-proxy-rules/0.log"
Jul 10 19:24:16.323363 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:16.323334    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd7ccc499-69tw6_66182fd1-a634-4db8-b35e-d64b8632eadc/kube-rbac-proxy-metrics/0.log"
Jul 10 19:24:17.791685 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.791656    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"]
Jul 10 19:24:17.792068 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.792036    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78501dc8-ebd7-44b4-8c24-fefc3fd67900" containerName="gather"
Jul 10 19:24:17.792068 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.792048    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="78501dc8-ebd7-44b4-8c24-fefc3fd67900" containerName="gather"
Jul 10 19:24:17.792068 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.792069    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78501dc8-ebd7-44b4-8c24-fefc3fd67900" containerName="copy"
Jul 10 19:24:17.792206 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.792094    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="78501dc8-ebd7-44b4-8c24-fefc3fd67900" containerName="copy"
Jul 10 19:24:17.792206 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.792156    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="78501dc8-ebd7-44b4-8c24-fefc3fd67900" containerName="copy"
Jul 10 19:24:17.792206 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.792166    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="78501dc8-ebd7-44b4-8c24-fefc3fd67900" containerName="gather"
Jul 10 19:24:17.795181 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.795164    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:17.798065 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.798047    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l54pc\"/\"openshift-service-ca.crt\""
Jul 10 19:24:17.798868 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.798849    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l54pc\"/\"kube-root-ca.crt\""
Jul 10 19:24:17.798978 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.798854    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l54pc\"/\"default-dockercfg-zmwvt\""
Jul 10 19:24:17.814935 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.814903    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"]
Jul 10 19:24:17.932460 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.932427    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0-proc\") pod \"perf-node-gather-daemonset-zbxbt\" (UID: \"0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:17.932460 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.932460    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq7k4\" (UniqueName: \"kubernetes.io/projected/0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0-kube-api-access-rq7k4\") pod \"perf-node-gather-daemonset-zbxbt\" (UID: \"0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:17.932674 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.932483    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0-podres\") pod \"perf-node-gather-daemonset-zbxbt\" (UID: \"0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:17.932674 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.932544    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0-lib-modules\") pod \"perf-node-gather-daemonset-zbxbt\" (UID: \"0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:17.932674 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.932592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0-sys\") pod \"perf-node-gather-daemonset-zbxbt\" (UID: \"0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:17.967425 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.967397    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/1.log"
Jul 10 19:24:17.973918 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:17.973888    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wfk4b_8cd32a9e-c8d4-4e46-8737-e786070a64f6/console-operator/2.log"
Jul 10 19:24:18.033564 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:18.033523    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0-sys\") pod \"perf-node-gather-daemonset-zbxbt\" (UID: \"0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:18.033711 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:18.033629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0-proc\") pod \"perf-node-gather-daemonset-zbxbt\" (UID: \"0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:18.033711 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:18.033648    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0-sys\") pod \"perf-node-gather-daemonset-zbxbt\" (UID: \"0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:18.033711 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:18.033652    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq7k4\" (UniqueName: \"kubernetes.io/projected/0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0-kube-api-access-rq7k4\") pod \"perf-node-gather-daemonset-zbxbt\" (UID: \"0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:18.033711 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:18.033702    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0-podres\") pod \"perf-node-gather-daemonset-zbxbt\" (UID: \"0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:18.033891 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:18.033734    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0-lib-modules\") pod \"perf-node-gather-daemonset-zbxbt\" (UID: \"0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:18.033891 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:18.033744    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0-proc\") pod \"perf-node-gather-daemonset-zbxbt\" (UID: \"0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:18.033891 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:18.033811    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0-podres\") pod \"perf-node-gather-daemonset-zbxbt\" (UID: \"0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:18.033891 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:18.033859    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0-lib-modules\") pod \"perf-node-gather-daemonset-zbxbt\" (UID: \"0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:18.043672 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:18.043611    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq7k4\" (UniqueName: \"kubernetes.io/projected/0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0-kube-api-access-rq7k4\") pod \"perf-node-gather-daemonset-zbxbt\" (UID: \"0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:18.104879 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:18.104817    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:18.243350 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:18.243323    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"]
Jul 10 19:24:18.245970 ip-10-0-135-91 kubenswrapper[2561]: W0710 19:24:18.245943    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0bac0a4f_89bc_44fc_9cd5_a4e02b07abf0.slice/crio-ba546cc92f07cba20aa01074424fb386d48fd11cf8ff8a29596ed79481620944 WatchSource:0}: Error finding container ba546cc92f07cba20aa01074424fb386d48fd11cf8ff8a29596ed79481620944: Status 404 returned error can't find the container with id ba546cc92f07cba20aa01074424fb386d48fd11cf8ff8a29596ed79481620944
Jul 10 19:24:18.506293 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:18.506263    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-5vtgc_2732d02d-0e1b-4073-aba7-e668344351c9/download-server/0.log"
Jul 10 19:24:18.915149 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:18.915110    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt" event={"ID":"0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0","Type":"ContainerStarted","Data":"afa59eeb5a7e8d7addff95e0d8b275b7191b9f3cbf448d942ff7fe557493d6af"}
Jul 10 19:24:18.915531 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:18.915172    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:18.915531 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:18.915187    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt" event={"ID":"0bac0a4f-89bc-44fc-9cd5-a4e02b07abf0","Type":"ContainerStarted","Data":"ba546cc92f07cba20aa01074424fb386d48fd11cf8ff8a29596ed79481620944"}
Jul 10 19:24:18.934977 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:18.934931    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt" podStartSLOduration=1.934915614 podStartE2EDuration="1.934915614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:24:18.932709471 +0000 UTC m=+2682.406802356" watchObservedRunningTime="2026-07-10 19:24:18.934915614 +0000 UTC m=+2682.409008498"
Jul 10 19:24:20.280231 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:20.280199    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tlhxg_560b6122-2684-461d-a4e2-34ef3351b1bc/dns/0.log"
Jul 10 19:24:20.302221 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:20.302195    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tlhxg_560b6122-2684-461d-a4e2-34ef3351b1bc/kube-rbac-proxy/0.log"
Jul 10 19:24:20.352065 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:20.352036    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nq2tz_d4473d8a-7b80-4430-9006-674467a52709/dns-node-resolver/0.log"
Jul 10 19:24:20.995286 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:20.995256    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2tjml_3f0e0679-451d-492b-a83b-8c7662d459a7/node-ca/0.log"
Jul 10 19:24:22.694658 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:22.694629    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cx9r7_91ec32f4-5c41-4dea-a301-97256b26976d/serve-healthcheck-canary/0.log"
Jul 10 19:24:23.196159 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:23.196137    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dbkh5_68b929ae-2f98-46da-b893-82064267e801/kube-rbac-proxy/0.log"
Jul 10 19:24:23.218197 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:23.218168    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dbkh5_68b929ae-2f98-46da-b893-82064267e801/exporter/0.log"
Jul 10 19:24:23.241803 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:23.241779    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dbkh5_68b929ae-2f98-46da-b893-82064267e801/extractor/0.log"
Jul 10 19:24:24.930826 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:24.930799    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-zbxbt"
Jul 10 19:24:26.203241 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:26.203213    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-849cc6f9d7-48b6z_ccda2068-656d-48a0-9f3d-293cf7124795/manager/0.log"
Jul 10 19:24:27.098672 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:27.098642    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-659578d74d-bkptw_2f9b899e-e2b5-4c68-a938-feaf6ed5e1e2/manager/0.log"
Jul 10 19:24:27.215047 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:27.215023    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-2488h_e3ef5227-5997-4a67-9e3a-c6c88b50d3ed/server/0.log"
Jul 10 19:24:27.425005 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:27.424968    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-ks2cj_03c1aff7-9cff-4589-9773-92b477ff7d9e/manager/0.log"
Jul 10 19:24:32.835309 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:32.835275    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-nwzj8_d1efc208-97e2-4f83-867c-084b1863357b/migrator/0.log"
Jul 10 19:24:32.856960 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:32.856926    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-nwzj8_d1efc208-97e2-4f83-867c-084b1863357b/graceful-termination/0.log"
Jul 10 19:24:33.262312 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:33.262278    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-v7qt9_27e2c236-1c19-48b0-ad1a-38d02e492b6c/kube-storage-version-migrator-operator/1.log"
Jul 10 19:24:33.263175 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:33.263153    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-v7qt9_27e2c236-1c19-48b0-ad1a-38d02e492b6c/kube-storage-version-migrator-operator/0.log"
Jul 10 19:24:34.339396 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:34.339372    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mskk8_338a7e27-acaf-4107-8de7-08dfcae53a00/kube-multus-additional-cni-plugins/0.log"
Jul 10 19:24:34.359798 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:34.359772    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mskk8_338a7e27-acaf-4107-8de7-08dfcae53a00/egress-router-binary-copy/0.log"
Jul 10 19:24:34.380925 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:34.380897    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mskk8_338a7e27-acaf-4107-8de7-08dfcae53a00/cni-plugins/0.log"
Jul 10 19:24:34.402349 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:34.402323    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mskk8_338a7e27-acaf-4107-8de7-08dfcae53a00/bond-cni-plugin/0.log"
Jul 10 19:24:34.422613 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:34.422581    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mskk8_338a7e27-acaf-4107-8de7-08dfcae53a00/routeoverride-cni/0.log"
Jul 10 19:24:34.443171 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:34.443140    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mskk8_338a7e27-acaf-4107-8de7-08dfcae53a00/whereabouts-cni-bincopy/0.log"
Jul 10 19:24:34.463977 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:34.463948    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mskk8_338a7e27-acaf-4107-8de7-08dfcae53a00/whereabouts-cni/0.log"
Jul 10 19:24:34.879190 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:34.879161    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qczm9_c22f9db6-8f10-42be-a34d-cf75c7272b60/kube-multus/0.log"
Jul 10 19:24:34.903983 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:34.903956    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4rggc_17b91ece-5833-48f6-95f1-7df776a892f8/network-metrics-daemon/0.log"
Jul 10 19:24:34.926294 ip-10-0-135-91 kubenswrapper[2561]: I0710 19:24:34.926271    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4rggc_17b91ece-5833-48f6-95f1-7df776a892f8/kube-rbac-proxy/0.log"