Jul 28 16:01:53.315297 ip-10-0-131-181 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 16:01:53.764154 ip-10-0-131-181 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 16:01:53.764154 ip-10-0-131-181 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 16:01:53.764154 ip-10-0-131-181 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 16:01:53.764154 ip-10-0-131-181 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 16:01:53.764154 ip-10-0-131-181 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 16:01:53.765166 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.765091    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 16:01:53.772465 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772450    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:01:53.772465 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772464    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772468    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772472    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772475    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772478    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772481    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772484    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772487    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772490    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772494    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772497    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772500    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772503    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772506    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772508    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772511    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772521    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772524    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772526    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:01:53.772534 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772529    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772532    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772534    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772537    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772540    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772542    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772545    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772548    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772551    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772553    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772556    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772558    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772561    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772563    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772566    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772569    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772571    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772574    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772577    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772580    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:01:53.772985 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772582    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772585    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772588    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772590    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772594    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772596    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772599    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772601    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772604    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772606    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772609    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772611    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772613    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772616    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772619    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772624    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772628    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772631    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772634    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772637    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:01:53.773512 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772640    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772642    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772645    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772648    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772651    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772653    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772656    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772659    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772663    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772666    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772669    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772672    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772674    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772677    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772680    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772682    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772685    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772687    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772692    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:01:53.774116 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772695    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772697    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772700    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772702    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772705    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772708    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772711    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772714    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772717    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772720    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772723    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772725    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772728    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772730    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772734    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772738    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772741    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772743    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772746    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:01:53.774574 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.772748    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773440    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773447    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773450    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773452    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773455    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773458    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773462    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773464    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773467    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773470    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773473    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773475    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773478    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773481    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773483    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773485    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773488    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773491    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773494    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:01:53.775084 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773499    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773502    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773505    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773508    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773511    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773514    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773516    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773519    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773522    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773524    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773527    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773530    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773532    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773535    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773538    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773540    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773543    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773545    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773548    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773553    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:01:53.775569 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773556    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773559    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773561    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773563    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773566    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773568    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773571    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773574    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773577    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773579    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773581    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773584    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773588    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773591    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773593    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773596    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773599    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773601    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773604    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773607    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:01:53.776088 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773609    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:01:53.776563 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773611    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:01:53.776563 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773614    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:01:53.776563 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773616    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:01:53.776563 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773621    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:01:53.776563 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773625    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:01:53.776563 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773628    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:01:53.776563 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773632    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:01:53.776563 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773634    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:01:53.776563 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773637    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:01:53.776563 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773639    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:01:53.776563 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773642    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:01:53.776563 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773645    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:01:53.776563 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773648    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:01:53.776563 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773650    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:01:53.776563 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773653    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:01:53.776563 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773655    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:01:53.776563 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773659    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:01:53.776563 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773663    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773665    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773668    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773671    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773674    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773676    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773680    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773683    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773685    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773688    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773690    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773694    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773697    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773700    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773703    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773706    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773709    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773712    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773714    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773717    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:01:53.776986 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773720    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.773723    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774089    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774094    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774097    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774100    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774103    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774106    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774109    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774111    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774114    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774117    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774119    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774123    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774125    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774128    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774131    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774134    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774138    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774140    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:01:53.777495 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774143    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774145    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774148    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774151    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774153    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774156    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774158    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774161    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774163    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774166    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774169    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774171    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774174    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774177    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774179    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774182    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774184    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774186    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774189    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774192    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:01:53.777965 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774195    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774197    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774200    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774202    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774205    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774207    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774212    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774215    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774218    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774221    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774223    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774226    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774229    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774231    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774234    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774236    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774239    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774243    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774246    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774249    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:01:53.778451 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774252    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774255    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774257    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774260    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774262    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774265    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774268    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774272    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774275    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774278    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774281    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774284    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774286    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774289    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774291    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774294    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774297    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774299    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774303    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:01:53.778937 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774307    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774310    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774316    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774318    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774321    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774324    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774327    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774329    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774332    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774334    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774337    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774339    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774342    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774344    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774347    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774349    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774352    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774355    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774357    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774360    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:01:53.779411 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774362    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.774365    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774426    2581 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774433    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774439    2581 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774443    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774449    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774454    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774460    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774465    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774470    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774475    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774478    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774482    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774486    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774489    2581 flags.go:64] FLAG: --cgroup-root=""
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774492    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774495    2581 flags.go:64] FLAG: --client-ca-file=""
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774498    2581 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774501    2581 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774505    2581 flags.go:64] FLAG: --cluster-domain=""
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774507    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774510    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774513    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 16:01:53.779887 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774517    2581 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774520    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774523    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774526    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774529    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774532    2581 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774535    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774538    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774541    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774543    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774548    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774551    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774553    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774556    2581 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774559    2581 flags.go:64] FLAG: --enable-server="true"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774563    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774567    2581 flags.go:64] FLAG: --event-burst="100"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774570    2581 flags.go:64] FLAG: --event-qps="50"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774573    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774577    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774580    2581 flags.go:64] FLAG: --eviction-hard=""
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774584    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774588    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774591    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774594    2581 flags.go:64] FLAG: --eviction-soft=""
Jul 28 16:01:53.780462 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774597    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774600    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774602    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774605    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774608    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774611    2581 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774614    2581 flags.go:64] FLAG: --feature-gates=""
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774617    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774620    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774623    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774626    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774630    2581 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774632    2581 flags.go:64] FLAG: --help="false"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774635    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-131-181.ec2.internal"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774638    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774641    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774644    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774647    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774650    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774653    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774656    2581 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774659    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774662    2581 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 16:01:53.781118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774665    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774668    2581 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774671    2581 flags.go:64] FLAG: --kube-reserved=""
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774675    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774678    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774681    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774685    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774688    2581 flags.go:64] FLAG: --lock-file=""
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774691    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774694    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774697    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774702    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774705    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774707    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774710    2581 flags.go:64] FLAG: --logging-format="text"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774713    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774716    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774719    2581 flags.go:64] FLAG: --manifest-url=""
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774722    2581 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774726    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774729    2581 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774732    2581 flags.go:64] FLAG: --max-pods="110"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774735    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774738    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774741    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 16:01:53.781775 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774744    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774747    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774750    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774753    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774759    2581 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774762    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774765    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774768    2581 flags.go:64] FLAG: --pod-cidr=""
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774771    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774778    2581 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774780    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774785    2581 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774788    2581 flags.go:64] FLAG: --port="10250"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774794    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774797    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bba8698d90ad211b"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774800    2581 flags.go:64] FLAG: --qos-reserved=""
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774803    2581 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774806    2581 flags.go:64] FLAG: --register-node="true"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774809    2581 flags.go:64] FLAG: --register-with-taints=""
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774820    2581 flags.go:64] FLAG: --registry-burst="10"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774823    2581 flags.go:64] FLAG: --registry-qps="5"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774826    2581 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774829    2581 flags.go:64] FLAG: --reserved-memory=""
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774832    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774835    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 16:01:53.782407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774838    2581 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774841    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774843    2581 flags.go:64] FLAG: --runonce="false"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774846    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774849    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774852    2581 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774855    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774858    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774861    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774864    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774867    2581 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774869    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774872    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774875    2581 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774878    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774881    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774884    2581 flags.go:64] FLAG: --system-cgroups=""
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774887    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774901    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774904    2581 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774909    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774912    2581 flags.go:64] FLAG: --tls-min-version=""
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774915    2581 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774918    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774921    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 16:01:53.783028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774924    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774927    2581 flags.go:64] FLAG: --v="2"
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774931    2581 flags.go:64] FLAG: --version="false"
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774935    2581 flags.go:64] FLAG: --vmodule=""
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774938    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.774941    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775077    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775081    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775085    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775088    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775091    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775094    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775097    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775100    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775103    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775105    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775108    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775111    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775114    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775116    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775119    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:01:53.783643 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775121    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775124    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775127    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775130    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775134    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775137    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775140    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775142    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775145    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775148    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775150    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775153    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775155    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775158    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775160    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775163    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775165    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775167    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775170    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775172    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:01:53.784166 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775175    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775177    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775180    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775182    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775185    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775187    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775190    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775192    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775195    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775197    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775200    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775202    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775205    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775207    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775210    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775212    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775216    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775219    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775222    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:01:53.784674 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775224    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775227    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775230    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775232    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775235    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775239    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775243    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775246    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775248    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775251    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775253    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775256    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775258    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775261    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775263    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775266    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775268    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775271    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775274    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:01:53.785164 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775277    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:01:53.785928 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775279    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:01:53.785928 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775283    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:01:53.785928 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775287    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:01:53.785928 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775290    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:01:53.785928 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775293    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:01:53.785928 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775295    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:01:53.785928 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775298    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:01:53.785928 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775301    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:01:53.785928 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775304    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:01:53.785928 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775308    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:01:53.785928 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775312    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:01:53.785928 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775315    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:01:53.785928 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775317    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:01:53.785928 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775320    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:01:53.785928 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775323    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:01:53.785928 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775326    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:01:53.785928 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775329    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:01:53.785928 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775331    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:01:53.786398 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775335    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:01:53.786398 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775338    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:01:53.786398 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775341    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:01:53.786398 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775344    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:01:53.786398 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775346    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:01:53.786398 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775349    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:01:53.786398 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.775352    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:01:53.786398 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.776073    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 16:01:53.786398 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.782951    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 16:01:53.786398 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.782964    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 16:01:53.786398 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.782977    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 16:01:53.786398 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.782983    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 16:01:53.786398 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783050    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:01:53.786398 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783055    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783058    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783062    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783065    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783068    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783071    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783074    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783077    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783079    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783082    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783084    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783087    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783090    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783092    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783095    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783098    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783101    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783103    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783106    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783108    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:01:53.786997 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783111    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783114    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783116    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783119    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783121    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783124    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783126    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783129    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783133    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783136    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783139    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783141    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783144    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783146    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783149    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783152    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783154    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783157    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783160    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783162    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:01:53.787795 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783165    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783167    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783170    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783172    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783175    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783177    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783180    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783183    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783185    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783189    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783193    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783195    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783198    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783202    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783205    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783207    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783210    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783213    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783216    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:01:53.788312 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783218    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783222    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783225    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783228    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783230    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783233    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783235    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783238    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783240    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783243    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783245    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783248    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783250    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783253    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783255    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783258    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783260    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783263    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783265    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783267    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:01:53.788888 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783270    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:01:53.789492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783272    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:01:53.789492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783276    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:01:53.789492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783280    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:01:53.789492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783282    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:01:53.789492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783285    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:01:53.789492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783287    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:01:53.789492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783290    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:01:53.789492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783292    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:01:53.789492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783296    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:01:53.789492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783299    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:01:53.789492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783302    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:01:53.789492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783305    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:01:53.789492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783308    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:01:53.789492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783311    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:01:53.789492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783314    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:01:53.789492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783316    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:01:53.789492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783319    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:01:53.789492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783322    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:01:53.789924 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.783326    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 16:01:53.789924 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783434    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:01:53.789924 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783438    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:01:53.789924 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783441    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:01:53.789924 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783444    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:01:53.789924 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783447    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:01:53.789924 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783449    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:01:53.789924 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783452    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:01:53.789924 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783455    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:01:53.789924 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783457    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:01:53.789924 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783460    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:01:53.789924 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783462    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:01:53.789924 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783464    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:01:53.789924 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783467    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:01:53.789924 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783469    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783472    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783474    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783477    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783479    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783482    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783484    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783486    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783489    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783491    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783494    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783496    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783499    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783502    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783504    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783508    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783510    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783512    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783515    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783518    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:01:53.790365 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783520    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783523    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783525    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783527    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783530    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783532    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783535    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783537    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783540    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783542    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783545    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783547    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783550    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783552    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783555    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783557    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783560    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783563    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783565    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783568    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:01:53.790842 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783570    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:01:53.791492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783573    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:01:53.791492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783575    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:01:53.791492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783577    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:01:53.791492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783580    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:01:53.791492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783584    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:01:53.791492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783586    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:01:53.791492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783588    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:01:53.791492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783593    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:01:53.791492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783596    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:01:53.791492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783598    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:01:53.791492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783601    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:01:53.791492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783603    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:01:53.791492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783606    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:01:53.791492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783609    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:01:53.791492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783613    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:01:53.791492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783617    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:01:53.791492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783619    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:01:53.791492 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783622    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783625    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783627    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783630    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783632    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783635    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783637    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783640    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783642    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783645    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783648    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783650    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783653    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783655    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783658    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783661    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783663    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783666    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783668    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783671    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:01:53.791927 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783673    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:01:53.792427 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783676    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:01:53.792427 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783678    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:01:53.792427 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783681    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:01:53.792427 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783684    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:01:53.792427 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783686    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:01:53.792427 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783689    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:01:53.792427 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:53.783692    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:01:53.792427 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.783697    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 16:01:53.792427 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.783788    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 16:01:53.792427 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.788425    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 16:01:53.792427 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.790198    2581 server.go:1019] "Starting client certificate rotation"
Jul 28 16:01:53.792427 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.790296    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 16:01:53.792427 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.790328    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 16:01:53.793115 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.793101    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 16:01:53.794192 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.794169    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 16:01:53.804756 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.804738    2581 log.go:25] "Validated CRI v1 runtime API"
Jul 28 16:01:53.808705 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.808690    2581 log.go:25] "Validated CRI v1 image API"
Jul 28 16:01:53.809476 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.809459    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 16:01:53.812281 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.812259    2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8267c6e8-69d5-4da7-abb8-fb853b5414bb:/dev/nvme0n1p3 a4c4d207-83ab-4bb4-8dc8-623c05745e0c:/dev/nvme0n1p4]
Jul 28 16:01:53.812353 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.812279    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 16:01:53.818282 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.818170    2581 manager.go:216] Machine: {Timestamp:2026-07-28 16:01:53.816750786 +0000 UTC m=+0.375307323 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3147216 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2944b96bacb241631e284ebc432df3 SystemUUID:ec2944b9-6bac-b241-631e-284ebc432df3 BootID:4253a8f1-e412-44c8-8153-6d4c1650fe47 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:01:f1:c2:f2:23 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:01:f1:c2:f2:23 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:71:8f:87:97:9a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 16:01:53.818282 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.818270    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 16:01:53.818420 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.818342    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 16:01:53.821812 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.821785    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 16:01:53.821944 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.821816    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-181.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 16:01:53.823369 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.823359    2581 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 16:01:53.823412 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.823371    2581 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 16:01:53.823412 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.823383    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 16:01:53.824235 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.824223    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 16:01:53.826157 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.826144    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 16:01:53.827634 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.827616    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 16:01:53.830922 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.830908    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 16:01:53.831042 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.831032    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 16:01:53.831091 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.831061    2581 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 16:01:53.831091 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.831070    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 16:01:53.831091 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.831083    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 16:01:53.831091 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.831092    2581 kubelet.go:398] "Adding apiserver pod source"
Jul 28 16:01:53.831219 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.831116    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 16:01:53.831476 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.831460    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 16:01:53.831513 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.831494    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 16:01:53.833825 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.833804    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 16:01:53.837266 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.837250    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 16:01:53.837367 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.837281    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 16:01:53.838170 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.838155    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 16:01:53.838221 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.838180    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 16:01:53.838221 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.838190    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 16:01:53.838221 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.838196    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 16:01:53.838221 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.838204    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 16:01:53.838221 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.838209    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 16:01:53.838221 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.838216    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 16:01:53.838221 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.838221    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 16:01:53.838395 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.838227    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 16:01:53.838395 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.838233    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 16:01:53.838395 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.838246    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 16:01:53.838395 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.838251    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 16:01:53.838395 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.838274    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 16:01:53.838395 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.838282    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 16:01:53.841339 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.841323    2581 server.go:1268] "Started kubelet"
Jul 28 16:01:53.841468 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.841401    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 16:01:53.841538 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.841428    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 16:01:53.841538 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.841494    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 16:01:53.842156 ip-10-0-131-181 systemd[1]: Started Kubernetes Kubelet.
Jul 28 16:01:53.843352 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.842835    2581 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 16:01:53.845051 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.844106    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 16:01:53.847685 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:53.847664    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 16:01:53.848904 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:53.847831    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-181.ec2.internal.18c67f858b2c563c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-181.ec2.internal,UID:ip-10-0-131-181.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-181.ec2.internal,},FirstTimestamp:2026-07-28 16:01:53.841296956 +0000 UTC m=+0.399853489,LastTimestamp:2026-07-28 16:01:53.841296956 +0000 UTC m=+0.399853489,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-181.ec2.internal,}"
Jul 28 16:01:53.852583 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.852566    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-181.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 16:01:53.852860 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:53.852840    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-181.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 16:01:53.852904 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:53.852839    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 16:01:53.854502 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.854489    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 16:01:53.854589 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.854504    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 16:01:53.854589 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:53.854563    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-181.ec2.internal\" not found"
Jul 28 16:01:53.855255 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.855236    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 16:01:53.855255 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.855239    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 16:01:53.855409 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.855273    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 16:01:53.855458 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.855409    2581 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 16:01:53.855458 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.855420    2581 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 16:01:53.855610 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.855596    2581 factory.go:153] Registering CRI-O factory
Jul 28 16:01:53.855683 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.855613    2581 factory.go:223] Registration of the crio container factory successfully
Jul 28 16:01:53.855683 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.855677    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 16:01:53.855772 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.855687    2581 factory.go:55] Registering systemd factory
Jul 28 16:01:53.855772 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.855697    2581 factory.go:223] Registration of the systemd container factory successfully
Jul 28 16:01:53.855772 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.855717    2581 factory.go:103] Registering Raw factory
Jul 28 16:01:53.855772 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.855731    2581 manager.go:1196] Started watching for new ooms in manager
Jul 28 16:01:53.856447 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.856232    2581 manager.go:349] Starting recovery of all containers
Jul 28 16:01:53.857678 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:53.857635    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-181.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 16:01:53.857767 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:53.857700    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 16:01:53.869683 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.869665    2581 manager.go:354] Recovery completed
Jul 28 16:01:53.875760 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.875739    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:01:53.877752 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.877733    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:01:53.877813 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.877767    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:01:53.877813 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.877782    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:01:53.878304 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.878281    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 16:01:53.878304 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.878299    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 16:01:53.878304 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.878302    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:01:53.878434 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.878316    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 16:01:53.879516 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.879498    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:01:53.879577 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.879530    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:01:53.879577 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.879540    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:01:53.879577 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:53.879505    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-181.ec2.internal.18c67f858d589991  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-181.ec2.internal,UID:ip-10-0-131-181.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-181.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-181.ec2.internal,},FirstTimestamp:2026-07-28 16:01:53.877752209 +0000 UTC m=+0.436308749,LastTimestamp:2026-07-28 16:01:53.877752209 +0000 UTC m=+0.436308749,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-181.ec2.internal,}"
Jul 28 16:01:53.880429 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.880409    2581 policy_none.go:49] "None policy: Start"
Jul 28 16:01:53.880429 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.880424    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 16:01:53.880512 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.880433    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 16:01:53.880847 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.880832    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cfc8q"
Jul 28 16:01:53.882442 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.882430    2581 policy_none.go:47] "Start"
Jul 28 16:01:53.890711 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.890690    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cfc8q"
Jul 28 16:01:53.893462 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:53.893382    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-181.ec2.internal.18c67f858d58eb8a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-181.ec2.internal,UID:ip-10-0-131-181.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-181.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-181.ec2.internal,},FirstTimestamp:2026-07-28 16:01:53.877773194 +0000 UTC m=+0.436329729,LastTimestamp:2026-07-28 16:01:53.877773194 +0000 UTC m=+0.436329729,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-181.ec2.internal,}"
Jul 28 16:01:53.907667 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.907653    2581 manager.go:337] "Starting Device Plugin manager"
Jul 28 16:01:53.932161 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:53.907687    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 16:01:53.932161 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.907699    2581 server.go:87] "Starting device plugin registration server"
Jul 28 16:01:53.932161 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.907925    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 16:01:53.932161 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.907938    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 16:01:53.932161 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.908036    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 16:01:53.932161 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.908107    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 16:01:53.932161 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.908116    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 16:01:53.932161 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:53.908585    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 16:01:53.932161 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:53.908627    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-181.ec2.internal\" not found"
Jul 28 16:01:53.975516 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.975487    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 16:01:53.976621 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.976603    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 16:01:53.976712 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.976636    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 16:01:53.976712 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.976654    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 16:01:53.976712 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:53.976689    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 16:01:53.979555 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:53.979537    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:01:54.008730 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.008716    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:01:54.009491 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.009474    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:01:54.009576 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.009504    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:01:54.009576 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.009514    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:01:54.009576 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.009534    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-181.ec2.internal"
Jul 28 16:01:54.019704 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.019664    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-181.ec2.internal"
Jul 28 16:01:54.019704 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:54.019684    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-181.ec2.internal\": node \"ip-10-0-131-181.ec2.internal\" not found"
Jul 28 16:01:54.034742 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:54.034724    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-181.ec2.internal\" not found"
Jul 28 16:01:54.077865 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.077718    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-181.ec2.internal"]
Jul 28 16:01:54.077941 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.077899    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:01:54.078688 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.078675    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:01:54.078741 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.078702    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:01:54.078741 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.078715    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:01:54.081073 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.081061    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:01:54.081196 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.081174    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal"
Jul 28 16:01:54.081234 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.081207    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:01:54.082406 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.082392    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:01:54.082475 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.082406    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:01:54.082475 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.082415    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:01:54.082475 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.082425    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:01:54.082475 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.082429    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:01:54.082475 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.082435    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:01:54.085118 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.085103    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-181.ec2.internal"
Jul 28 16:01:54.085160 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.085131    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:01:54.085795 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.085777    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:01:54.085877 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.085807    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:01:54.085877 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.085821    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:01:54.101507 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:54.101488    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-181.ec2.internal\" not found" node="ip-10-0-131-181.ec2.internal"
Jul 28 16:01:54.105243 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:54.105228    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-181.ec2.internal\" not found" node="ip-10-0-131-181.ec2.internal"
Jul 28 16:01:54.135506 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:54.135488    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-181.ec2.internal\" not found"
Jul 28 16:01:54.156852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.156835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bfce947cf24c76d49863c92e16d2c2ef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal\" (UID: \"bfce947cf24c76d49863c92e16d2c2ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal"
Jul 28 16:01:54.156906 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.156857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfce947cf24c76d49863c92e16d2c2ef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal\" (UID: \"bfce947cf24c76d49863c92e16d2c2ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal"
Jul 28 16:01:54.156906 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.156873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e97bc9874712512bf8be887ba1ad9d3d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-181.ec2.internal\" (UID: \"e97bc9874712512bf8be887ba1ad9d3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-181.ec2.internal"
Jul 28 16:01:54.236140 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:54.236114    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-181.ec2.internal\" not found"
Jul 28 16:01:54.257438 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.257418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bfce947cf24c76d49863c92e16d2c2ef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal\" (UID: \"bfce947cf24c76d49863c92e16d2c2ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal"
Jul 28 16:01:54.257506 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.257443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfce947cf24c76d49863c92e16d2c2ef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal\" (UID: \"bfce947cf24c76d49863c92e16d2c2ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal"
Jul 28 16:01:54.257506 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.257460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e97bc9874712512bf8be887ba1ad9d3d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-181.ec2.internal\" (UID: \"e97bc9874712512bf8be887ba1ad9d3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-181.ec2.internal"
Jul 28 16:01:54.257506 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.257501    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e97bc9874712512bf8be887ba1ad9d3d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-181.ec2.internal\" (UID: \"e97bc9874712512bf8be887ba1ad9d3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-181.ec2.internal"
Jul 28 16:01:54.257606 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.257509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bfce947cf24c76d49863c92e16d2c2ef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal\" (UID: \"bfce947cf24c76d49863c92e16d2c2ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal"
Jul 28 16:01:54.257606 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.257512    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfce947cf24c76d49863c92e16d2c2ef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal\" (UID: \"bfce947cf24c76d49863c92e16d2c2ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal"
Jul 28 16:01:54.336854 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:54.336801    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-181.ec2.internal\" not found"
Jul 28 16:01:54.404334 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.404310    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal"
Jul 28 16:01:54.408083 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.408066    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-181.ec2.internal"
Jul 28 16:01:54.437247 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:54.437228    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-181.ec2.internal\" not found"
Jul 28 16:01:54.537758 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:54.537734    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-181.ec2.internal\" not found"
Jul 28 16:01:54.638292 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:54.638244    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-181.ec2.internal\" not found"
Jul 28 16:01:54.738813 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:54.738791    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-181.ec2.internal\" not found"
Jul 28 16:01:54.788796 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.788770    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 16:01:54.789166 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.788913    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 16:01:54.839128 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:54.839104    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-181.ec2.internal\" not found"
Jul 28 16:01:54.844768 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.844747    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:01:54.897530 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.897501    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 15:56:54 +0000 UTC" deadline="2028-04-24 11:57:16.098167938 +0000 UTC"
Jul 28 16:01:54.897530 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.897529    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15259h55m21.200642006s"
Jul 28 16:01:54.938071 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:54.938046    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfce947cf24c76d49863c92e16d2c2ef.slice/crio-44bd2b98caebc26116a2d9cb697d0a923026a3ed339346fbd27f8964f2094755 WatchSource:0}: Error finding container 44bd2b98caebc26116a2d9cb697d0a923026a3ed339346fbd27f8964f2094755: Status 404 returned error can't find the container with id 44bd2b98caebc26116a2d9cb697d0a923026a3ed339346fbd27f8964f2094755
Jul 28 16:01:54.939204 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:54.939181    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-181.ec2.internal\" not found"
Jul 28 16:01:54.943740 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.943724    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:01:54.948839 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:54.948811    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97bc9874712512bf8be887ba1ad9d3d.slice/crio-b8e8475e546f6c26ebb1ca8c9b92dd43126d120e4b02e1a17a4e3c30937f7c51 WatchSource:0}: Error finding container b8e8475e546f6c26ebb1ca8c9b92dd43126d120e4b02e1a17a4e3c30937f7c51: Status 404 returned error can't find the container with id b8e8475e546f6c26ebb1ca8c9b92dd43126d120e4b02e1a17a4e3c30937f7c51
Jul 28 16:01:54.960358 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.960331    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:01:54.961358 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.961343    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:01:54.961430 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.961370    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:01:54.961430 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.961383    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:01:54.982978 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.982936    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal" event={"ID":"bfce947cf24c76d49863c92e16d2c2ef","Type":"ContainerStarted","Data":"44bd2b98caebc26116a2d9cb697d0a923026a3ed339346fbd27f8964f2094755"}
Jul 28 16:01:54.984060 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.984014    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-181.ec2.internal" event={"ID":"e97bc9874712512bf8be887ba1ad9d3d","Type":"ContainerStarted","Data":"b8e8475e546f6c26ebb1ca8c9b92dd43126d120e4b02e1a17a4e3c30937f7c51"}
Jul 28 16:01:54.992490 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:54.992476    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:01:55.007756 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.007734    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:01:55.054852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.054828    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal"
Jul 28 16:01:55.071113 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.071093    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 16:01:55.072179 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.072165    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-181.ec2.internal"
Jul 28 16:01:55.081739 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.081719    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 16:01:55.832877 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.832846    2581 apiserver.go:52] "Watching apiserver"
Jul 28 16:01:55.841172 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.840871    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 16:01:55.841415 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.841382    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-zf8vz","kube-system/konnectivity-agent-r4p2b","kube-system/kube-apiserver-proxy-ip-10-0-131-181.ec2.internal","openshift-image-registry/node-ca-rwvsw","openshift-multus/multus-additional-cni-plugins-nnbz6","openshift-multus/network-metrics-daemon-zf8jb","openshift-network-diagnostics/network-check-target-4hp9r","openshift-network-operator/iptables-alerter-nkcdf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht","openshift-cluster-node-tuning-operator/tuned-mrvh4","openshift-dns/node-resolver-9b4sm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal","openshift-multus/multus-44lcg","openshift-ovn-kubernetes/ovnkube-node-khj85"]
Jul 28 16:01:55.844612 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.844173    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.846807 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.846589    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 16:01:55.846807 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.846608    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 16:01:55.846807 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.846686    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fx55l\""
Jul 28 16:01:55.846807 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.846801    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:01:55.847104 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:55.846876    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf8jb" podUID="c958899e-cac6-4e06-bf0a-e510c503c24d"
Jul 28 16:01:55.850379 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.850356    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.852644 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.852616    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rwvsw"
Jul 28 16:01:55.857199 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.855171    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.857199 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.855345    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 16:01:55.857199 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.855733    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 16:01:55.857199 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.855971    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 16:01:55.857199 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.856011    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 16:01:55.857199 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.856179    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 16:01:55.857199 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.856275    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5j8jv\""
Jul 28 16:01:55.857199 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.856387    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dxrtv\""
Jul 28 16:01:55.857199 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.856525    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 16:01:55.857199 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.856582    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 16:01:55.857771 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.857752    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4pths\""
Jul 28 16:01:55.858048 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.858025    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.858231 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.858098    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 16:01:55.858371 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.858351    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 16:01:55.858446 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.858351    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 16:01:55.858503 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.858362    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 16:01:55.860361 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.860236    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:01:55.860361 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:55.860297    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hp9r" podUID="92d984e8-b95b-4433-9a57-b5e15980a650"
Jul 28 16:01:55.860631 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.860605    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qqwmz\""
Jul 28 16:01:55.861440 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.860865    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 16:01:55.861440 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.861264    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 16:01:55.862884 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.862864    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r4p2b"
Jul 28 16:01:55.865147 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.864996    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bhgd6\""
Jul 28 16:01:55.865245 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865199    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 16:01:55.865455 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865430    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzxfb\" (UniqueName: \"kubernetes.io/projected/b400f70a-d50b-4eec-b1a4-fe911116339b-kube-api-access-jzxfb\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.865546 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a57f72c8-1133-4849-93f6-caa2d46580f4-os-release\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.865546 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-sysconfig\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.865546 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865518    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-run\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.865546 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l2fn\" (UniqueName: \"kubernetes.io/projected/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-kube-api-access-5l2fn\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.865772 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865567    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs\") pod \"network-metrics-daemon-zf8jb\" (UID: \"c958899e-cac6-4e06-bf0a-e510c503c24d\") " pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:01:55.865772 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865609    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-system-cni-dir\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.865772 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b400f70a-d50b-4eec-b1a4-fe911116339b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.865772 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4d7a739-0ea4-4d89-8fd4-f302b556a891-host\") pod \"node-ca-rwvsw\" (UID: \"f4d7a739-0ea4-4d89-8fd4-f302b556a891\") " pod="openshift-image-registry/node-ca-rwvsw"
Jul 28 16:01:55.865772 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-cnibin\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.865772 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-os-release\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.866064 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865772    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-multus-socket-dir-parent\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.866064 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865798    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-host-var-lib-cni-bin\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.866064 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865821    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4d7a739-0ea4-4d89-8fd4-f302b556a891-serviceca\") pod \"node-ca-rwvsw\" (UID: \"f4d7a739-0ea4-4d89-8fd4-f302b556a891\") " pod="openshift-image-registry/node-ca-rwvsw"
Jul 28 16:01:55.866064 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-host-var-lib-cni-multus\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.866064 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf9kk\" (UniqueName: \"kubernetes.io/projected/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-kube-api-access-gf9kk\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.866064 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865836    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 16:01:55.866064 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a57f72c8-1133-4849-93f6-caa2d46580f4-cnibin\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.866064 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865962    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a57f72c8-1133-4849-93f6-caa2d46580f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.866064 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865987    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:01:55.866064 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.865995    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lvqm\" (UniqueName: \"kubernetes.io/projected/a57f72c8-1133-4849-93f6-caa2d46580f4-kube-api-access-2lvqm\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.866064 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-modprobe-d\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.866064 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:55.866055    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf8vz" podUID="0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4"
Jul 28 16:01:55.866669 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-multus-conf-dir\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.866669 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-multus-daemon-config\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.866669 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866155    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b400f70a-d50b-4eec-b1a4-fe911116339b-device-dir\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.866669 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b400f70a-d50b-4eec-b1a4-fe911116339b-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.866669 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866204    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b400f70a-d50b-4eec-b1a4-fe911116339b-sys-fs\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.866669 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866227    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z87t9\" (UniqueName: \"kubernetes.io/projected/f4d7a739-0ea4-4d89-8fd4-f302b556a891-kube-api-access-z87t9\") pod \"node-ca-rwvsw\" (UID: \"f4d7a739-0ea4-4d89-8fd4-f302b556a891\") " pod="openshift-image-registry/node-ca-rwvsw"
Jul 28 16:01:55.866669 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866251    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-multus-cni-dir\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.866669 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-kubernetes\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.866669 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-host-var-lib-kubelet\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.866669 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-host-run-multus-certs\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.866669 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866395    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-etc-kubernetes\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.866669 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a57f72c8-1133-4849-93f6-caa2d46580f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.866669 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866446    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-sysctl-d\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.866669 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866470    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-sysctl-conf\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.866669 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866507    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-sys\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.866669 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-lib-modules\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.867407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-var-lib-kubelet\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.867407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-host\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.867407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-tuned\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.867407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866657    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rczzx\" (UniqueName: \"kubernetes.io/projected/c958899e-cac6-4e06-bf0a-e510c503c24d-kube-api-access-rczzx\") pod \"network-metrics-daemon-zf8jb\" (UID: \"c958899e-cac6-4e06-bf0a-e510c503c24d\") " pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:01:55.867407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b400f70a-d50b-4eec-b1a4-fe911116339b-socket-dir\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.867407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866729    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b400f70a-d50b-4eec-b1a4-fe911116339b-registration-dir\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.867407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866754    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-hostroot\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.867407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866800    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-systemd\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.867407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866839    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-host-run-netns\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.867407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866877    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-cni-binary-copy\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.867407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866903    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-host-run-k8s-cni-cncf-io\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.867407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866930    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a57f72c8-1133-4849-93f6-caa2d46580f4-system-cni-dir\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.867407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a57f72c8-1133-4849-93f6-caa2d46580f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.867407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.866979    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a57f72c8-1133-4849-93f6-caa2d46580f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.867407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.867004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-tmp\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.868388 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.868365    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9b4sm"
Jul 28 16:01:55.870395 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.870371    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 16:01:55.870601 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.870581    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-876sq\""
Jul 28 16:01:55.870675 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.870645    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 16:01:55.871499 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.871482    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nkcdf"
Jul 28 16:01:55.871986 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.871961    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 16:01:55.873422 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.873400    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 16:01:55.873853 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.873833    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 16:01:55.874221 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.874202    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6882q\""
Jul 28 16:01:55.874409 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.874389    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 16:01:55.874682 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.874654    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.878485 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.877900    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 16:01:55.878485 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.877915    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 16:01:55.878485 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.878106    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 16:01:55.878485 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.878228    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jwqgk\""
Jul 28 16:01:55.881734 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.880402    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 16:01:55.881734 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.880567    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 16:01:55.881734 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.881399    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 16:01:55.897912 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.897894    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ns5j9"
Jul 28 16:01:55.905361 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.905340    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ns5j9"
Jul 28 16:01:55.957249 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.957227    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 16:01:55.967295 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967270    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-log-socket\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.967418 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967312    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-system-cni-dir\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.967418 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b400f70a-d50b-4eec-b1a4-fe911116339b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.967418 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967361    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4d7a739-0ea4-4d89-8fd4-f302b556a891-host\") pod \"node-ca-rwvsw\" (UID: \"f4d7a739-0ea4-4d89-8fd4-f302b556a891\") " pod="openshift-image-registry/node-ca-rwvsw"
Jul 28 16:01:55.967418 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-os-release\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.967418 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-host-var-lib-cni-bin\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.967719 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf9kk\" (UniqueName: \"kubernetes.io/projected/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-kube-api-access-gf9kk\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.967719 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lvqm\" (UniqueName: \"kubernetes.io/projected/a57f72c8-1133-4849-93f6-caa2d46580f4-kube-api-access-2lvqm\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.967719 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-kubelet\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.967719 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4d7a739-0ea4-4d89-8fd4-f302b556a891-serviceca\") pod \"node-ca-rwvsw\" (UID: \"f4d7a739-0ea4-4d89-8fd4-f302b556a891\") " pod="openshift-image-registry/node-ca-rwvsw"
Jul 28 16:01:55.967719 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a57f72c8-1133-4849-93f6-caa2d46580f4-cnibin\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.967719 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.967719 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d724302e-ab2b-46d7-ac99-0ebf76711d64-ovnkube-config\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.967719 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/201d9cf5-adcd-49dd-be6f-f8e20cedce09-agent-certs\") pod \"konnectivity-agent-r4p2b\" (UID: \"201d9cf5-adcd-49dd-be6f-f8e20cedce09\") " pod="kube-system/konnectivity-agent-r4p2b"
Jul 28 16:01:55.967719 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvr7k\" (UniqueName: \"kubernetes.io/projected/441a3269-baa8-4575-99a6-007b21792801-kube-api-access-kvr7k\") pod \"node-resolver-9b4sm\" (UID: \"441a3269-baa8-4575-99a6-007b21792801\") " pod="openshift-dns/node-resolver-9b4sm"
Jul 28 16:01:55.967719 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967671    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-multus-conf-dir\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.967719 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967699    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b400f70a-d50b-4eec-b1a4-fe911116339b-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.968324 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b400f70a-d50b-4eec-b1a4-fe911116339b-sys-fs\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.968324 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-etc-openvswitch\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.968324 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-cni-bin\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.968324 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d724302e-ab2b-46d7-ac99-0ebf76711d64-env-overrides\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.968324 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-etc-kubernetes\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.968324 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-sysctl-d\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.968324 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967902    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-sysctl-conf\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.968324 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967918    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a57f72c8-1133-4849-93f6-caa2d46580f4-cnibin\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.968324 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967930    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-run-ovn\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.968324 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967961    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-var-lib-kubelet\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.968324 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.967987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-host\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.968324 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-tuned\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.968324 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-system-cni-dir\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.968324 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968050    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t5wz\" (UniqueName: \"kubernetes.io/projected/d724302e-ab2b-46d7-ac99-0ebf76711d64-kube-api-access-6t5wz\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.968324 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b400f70a-d50b-4eec-b1a4-fe911116339b-socket-dir\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.968324 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968103    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b400f70a-d50b-4eec-b1a4-fe911116339b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.968324 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b400f70a-d50b-4eec-b1a4-fe911116339b-registration-dir\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.969191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968146    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4d7a739-0ea4-4d89-8fd4-f302b556a891-host\") pod \"node-ca-rwvsw\" (UID: \"f4d7a739-0ea4-4d89-8fd4-f302b556a891\") " pod="openshift-image-registry/node-ca-rwvsw"
Jul 28 16:01:55.969191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-hostroot\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.969191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-systemd\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.969191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-run-systemd\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.969191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968210    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-os-release\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.969191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-cni-netd\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.969191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-host-var-lib-cni-bin\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.969191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/441a3269-baa8-4575-99a6-007b21792801-tmp-dir\") pod \"node-resolver-9b4sm\" (UID: \"441a3269-baa8-4575-99a6-007b21792801\") " pod="openshift-dns/node-resolver-9b4sm"
Jul 28 16:01:55.969191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-cni-binary-copy\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.969191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-host-run-k8s-cni-cncf-io\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.969191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a57f72c8-1133-4849-93f6-caa2d46580f4-system-cni-dir\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.969191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a57f72c8-1133-4849-93f6-caa2d46580f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.969191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-run-openvswitch\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.969191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968449    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzxfb\" (UniqueName: \"kubernetes.io/projected/b400f70a-d50b-4eec-b1a4-fe911116339b-kube-api-access-jzxfb\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.969191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a57f72c8-1133-4849-93f6-caa2d46580f4-os-release\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.969191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l2fn\" (UniqueName: \"kubernetes.io/projected/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-kube-api-access-5l2fn\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.969191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-var-lib-openvswitch\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.969993 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-node-log\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.969993 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-kubelet-config\") pod \"global-pull-secret-syncer-zf8vz\" (UID: \"0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4\") " pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:01:55.969993 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968649    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f2c74ea-8d67-407e-b213-adc356ab0a50-iptables-alerter-script\") pod \"iptables-alerter-nkcdf\" (UID: \"4f2c74ea-8d67-407e-b213-adc356ab0a50\") " pod="openshift-network-operator/iptables-alerter-nkcdf"
Jul 28 16:01:55.969993 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968677    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f2c74ea-8d67-407e-b213-adc356ab0a50-host-slash\") pod \"iptables-alerter-nkcdf\" (UID: \"4f2c74ea-8d67-407e-b213-adc356ab0a50\") " pod="openshift-network-operator/iptables-alerter-nkcdf"
Jul 28 16:01:55.969993 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968705    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-cnibin\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.969993 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-multus-socket-dir-parent\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.969993 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968762    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-host-var-lib-cni-multus\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.969993 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a57f72c8-1133-4849-93f6-caa2d46580f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.969993 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-modprobe-d\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.969993 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-run-ovn-kubernetes\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.969993 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968848    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-multus-conf-dir\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.969993 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-dbus\") pod \"global-pull-secret-syncer-zf8vz\" (UID: \"0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4\") " pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:01:55.969993 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968902    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-multus-daemon-config\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.969993 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b400f70a-d50b-4eec-b1a4-fe911116339b-device-dir\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.969993 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z87t9\" (UniqueName: \"kubernetes.io/projected/f4d7a739-0ea4-4d89-8fd4-f302b556a891-kube-api-access-z87t9\") pod \"node-ca-rwvsw\" (UID: \"f4d7a739-0ea4-4d89-8fd4-f302b556a891\") " pod="openshift-image-registry/node-ca-rwvsw"
Jul 28 16:01:55.969993 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968969    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b400f70a-d50b-4eec-b1a4-fe911116339b-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.969993 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-multus-cni-dir\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.970893 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.968997    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4d7a739-0ea4-4d89-8fd4-f302b556a891-serviceca\") pod \"node-ca-rwvsw\" (UID: \"f4d7a739-0ea4-4d89-8fd4-f302b556a891\") " pod="openshift-image-registry/node-ca-rwvsw"
Jul 28 16:01:55.970893 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.969016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-kubernetes\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.970893 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.969023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b400f70a-d50b-4eec-b1a4-fe911116339b-sys-fs\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.970893 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.969041    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-sys\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.970893 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.969129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-etc-kubernetes\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.970893 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.969154    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-sys\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.970893 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.969163    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-systemd-units\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.970893 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.969198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-run-netns\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.970893 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.969216    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-systemd\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.970893 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.969232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-host-var-lib-kubelet\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.970893 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.969266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-host-run-multus-certs\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.970893 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.969463    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-hostroot\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.970893 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.969551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-kubernetes\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.970893 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.969649    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-sysctl-d\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.970893 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.969754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-multus-cni-dir\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.970893 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.969768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-sysctl-conf\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.970893 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.969823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a57f72c8-1133-4849-93f6-caa2d46580f4-os-release\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.970893 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a57f72c8-1133-4849-93f6-caa2d46580f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.971852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-host-var-lib-kubelet\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.971852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970131    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b400f70a-d50b-4eec-b1a4-fe911116339b-socket-dir\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.971852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-multus-socket-dir-parent\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.971852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970190    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b400f70a-d50b-4eec-b1a4-fe911116339b-registration-dir\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.971852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970212    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-host-var-lib-cni-multus\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.971852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970237    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 16:01:55.971852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970295    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-modprobe-d\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.971852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970242    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-cnibin\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.971852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a57f72c8-1133-4849-93f6-caa2d46580f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.971852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970460    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl72p\" (UniqueName: \"kubernetes.io/projected/4f2c74ea-8d67-407e-b213-adc356ab0a50-kube-api-access-vl72p\") pod \"iptables-alerter-nkcdf\" (UID: \"4f2c74ea-8d67-407e-b213-adc356ab0a50\") " pod="openshift-network-operator/iptables-alerter-nkcdf"
Jul 28 16:01:55.971852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-lib-modules\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.971852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczzx\" (UniqueName: \"kubernetes.io/projected/c958899e-cac6-4e06-bf0a-e510c503c24d-kube-api-access-rczzx\") pod \"network-metrics-daemon-zf8jb\" (UID: \"c958899e-cac6-4e06-bf0a-e510c503c24d\") " pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:01:55.971852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970548    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d724302e-ab2b-46d7-ac99-0ebf76711d64-ovn-node-metrics-cert\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.971852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p6kb\" (UniqueName: \"kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb\") pod \"network-check-target-4hp9r\" (UID: \"92d984e8-b95b-4433-9a57-b5e15980a650\") " pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:01:55.971852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970609    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret\") pod \"global-pull-secret-syncer-zf8vz\" (UID: \"0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4\") " pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:01:55.971852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-slash\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.971852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d724302e-ab2b-46d7-ac99-0ebf76711d64-ovnkube-script-lib\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:55.972800 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/441a3269-baa8-4575-99a6-007b21792801-hosts-file\") pod \"node-resolver-9b4sm\" (UID: \"441a3269-baa8-4575-99a6-007b21792801\") " pod="openshift-dns/node-resolver-9b4sm"
Jul 28 16:01:55.972800 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970741    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-host-run-netns\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.972800 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a57f72c8-1133-4849-93f6-caa2d46580f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.972800 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-tmp\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.972800 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/201d9cf5-adcd-49dd-be6f-f8e20cedce09-konnectivity-ca\") pod \"konnectivity-agent-r4p2b\" (UID: \"201d9cf5-adcd-49dd-be6f-f8e20cedce09\") " pod="kube-system/konnectivity-agent-r4p2b"
Jul 28 16:01:55.972800 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970815    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-cni-binary-copy\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.972800 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-sysconfig\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.972800 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-run\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.972800 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970868    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-host-run-k8s-cni-cncf-io\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.972800 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970874    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs\") pod \"network-metrics-daemon-zf8jb\" (UID: \"c958899e-cac6-4e06-bf0a-e510c503c24d\") " pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:01:55.972800 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970928    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-var-lib-kubelet\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.972800 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a57f72c8-1133-4849-93f6-caa2d46580f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.972800 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:55.970984    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:55.972800 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.971036    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b400f70a-d50b-4eec-b1a4-fe911116339b-device-dir\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.972800 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:55.971042    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs podName:c958899e-cac6-4e06-bf0a-e510c503c24d nodeName:}" failed. No retries permitted until 2026-07-28 16:01:56.471022377 +0000 UTC m=+3.029578902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs") pod "network-metrics-daemon-zf8jb" (UID: "c958899e-cac6-4e06-bf0a-e510c503c24d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:55.972800 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.971083    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a57f72c8-1133-4849-93f6-caa2d46580f4-system-cni-dir\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.972800 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.971117    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-host-run-netns\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.973729 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.971134    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a57f72c8-1133-4849-93f6-caa2d46580f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.973729 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.971160    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-sysconfig\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.973729 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.971175    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-run\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.973729 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.971211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-host-run-multus-certs\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.973729 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.970985    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-host\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.973729 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.971576    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-multus-daemon-config\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.973729 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.971689    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-lib-modules\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.973729 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.971832    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a57f72c8-1133-4849-93f6-caa2d46580f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.974484 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.974447    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-etc-tuned\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.974484 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.974509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-tmp\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.976381 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.976361    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:01:55.979990 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.979928    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z87t9\" (UniqueName: \"kubernetes.io/projected/f4d7a739-0ea4-4d89-8fd4-f302b556a891-kube-api-access-z87t9\") pod \"node-ca-rwvsw\" (UID: \"f4d7a739-0ea4-4d89-8fd4-f302b556a891\") " pod="openshift-image-registry/node-ca-rwvsw"
Jul 28 16:01:55.980558 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.980537    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lvqm\" (UniqueName: \"kubernetes.io/projected/a57f72c8-1133-4849-93f6-caa2d46580f4-kube-api-access-2lvqm\") pod \"multus-additional-cni-plugins-nnbz6\" (UID: \"a57f72c8-1133-4849-93f6-caa2d46580f4\") " pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:55.981227 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.981203    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzxfb\" (UniqueName: \"kubernetes.io/projected/b400f70a-d50b-4eec-b1a4-fe911116339b-kube-api-access-jzxfb\") pod \"aws-ebs-csi-driver-node-vzxht\" (UID: \"b400f70a-d50b-4eec-b1a4-fe911116339b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:55.983255 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.983233    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf9kk\" (UniqueName: \"kubernetes.io/projected/ce86f1d2-93dd-4dde-9cda-f3e53fbd7642-kube-api-access-gf9kk\") pod \"multus-44lcg\" (UID: \"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642\") " pod="openshift-multus/multus-44lcg"
Jul 28 16:01:55.986621 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.986596    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l2fn\" (UniqueName: \"kubernetes.io/projected/9d2e4031-a6e4-47a3-b57c-6767de81ffd9-kube-api-access-5l2fn\") pod \"tuned-mrvh4\" (UID: \"9d2e4031-a6e4-47a3-b57c-6767de81ffd9\") " pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:55.987711 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:55.987689    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rczzx\" (UniqueName: \"kubernetes.io/projected/c958899e-cac6-4e06-bf0a-e510c503c24d-kube-api-access-rczzx\") pod \"network-metrics-daemon-zf8jb\" (UID: \"c958899e-cac6-4e06-bf0a-e510c503c24d\") " pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:01:56.071638 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.071603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-run-ovn\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.071796 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.071654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t5wz\" (UniqueName: \"kubernetes.io/projected/d724302e-ab2b-46d7-ac99-0ebf76711d64-kube-api-access-6t5wz\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.071796 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.071740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-run-ovn\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.071921 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.071803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-run-systemd\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.071921 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.071841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-cni-netd\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.071921 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.071869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/441a3269-baa8-4575-99a6-007b21792801-tmp-dir\") pod \"node-resolver-9b4sm\" (UID: \"441a3269-baa8-4575-99a6-007b21792801\") " pod="openshift-dns/node-resolver-9b4sm"
Jul 28 16:01:56.071921 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.071892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-run-systemd\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.071921 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.071916    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-cni-netd\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.072180 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.071918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-run-openvswitch\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.072180 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.071960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-run-openvswitch\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.072180 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.071976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-var-lib-openvswitch\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.072180 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-node-log\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.072180 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-kubelet-config\") pod \"global-pull-secret-syncer-zf8vz\" (UID: \"0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4\") " pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:01:56.072180 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072060    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-var-lib-openvswitch\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.072180 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072083    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f2c74ea-8d67-407e-b213-adc356ab0a50-iptables-alerter-script\") pod \"iptables-alerter-nkcdf\" (UID: \"4f2c74ea-8d67-407e-b213-adc356ab0a50\") " pod="openshift-network-operator/iptables-alerter-nkcdf"
Jul 28 16:01:56.072180 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072089    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-node-log\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.072180 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f2c74ea-8d67-407e-b213-adc356ab0a50-host-slash\") pod \"iptables-alerter-nkcdf\" (UID: \"4f2c74ea-8d67-407e-b213-adc356ab0a50\") " pod="openshift-network-operator/iptables-alerter-nkcdf"
Jul 28 16:01:56.072180 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072122    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-kubelet-config\") pod \"global-pull-secret-syncer-zf8vz\" (UID: \"0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4\") " pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:01:56.072180 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072140    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-run-ovn-kubernetes\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.072180 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-dbus\") pod \"global-pull-secret-syncer-zf8vz\" (UID: \"0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4\") " pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:01:56.072767 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f2c74ea-8d67-407e-b213-adc356ab0a50-host-slash\") pod \"iptables-alerter-nkcdf\" (UID: \"4f2c74ea-8d67-407e-b213-adc356ab0a50\") " pod="openshift-network-operator/iptables-alerter-nkcdf"
Jul 28 16:01:56.072767 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072200    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-systemd-units\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.072767 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072228    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/441a3269-baa8-4575-99a6-007b21792801-tmp-dir\") pod \"node-resolver-9b4sm\" (UID: \"441a3269-baa8-4575-99a6-007b21792801\") " pod="openshift-dns/node-resolver-9b4sm"
Jul 28 16:01:56.072767 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072232    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-systemd-units\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.072767 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072220    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-run-ovn-kubernetes\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.072767 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-run-netns\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.072767 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl72p\" (UniqueName: \"kubernetes.io/projected/4f2c74ea-8d67-407e-b213-adc356ab0a50-kube-api-access-vl72p\") pod \"iptables-alerter-nkcdf\" (UID: \"4f2c74ea-8d67-407e-b213-adc356ab0a50\") " pod="openshift-network-operator/iptables-alerter-nkcdf"
Jul 28 16:01:56.072767 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072266    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-dbus\") pod \"global-pull-secret-syncer-zf8vz\" (UID: \"0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4\") " pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:01:56.072767 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-run-netns\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.073196 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072994    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f2c74ea-8d67-407e-b213-adc356ab0a50-iptables-alerter-script\") pod \"iptables-alerter-nkcdf\" (UID: \"4f2c74ea-8d67-407e-b213-adc356ab0a50\") " pod="openshift-network-operator/iptables-alerter-nkcdf"
Jul 28 16:01:56.073196 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.072999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d724302e-ab2b-46d7-ac99-0ebf76711d64-ovn-node-metrics-cert\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.073196 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.073073    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p6kb\" (UniqueName: \"kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb\") pod \"network-check-target-4hp9r\" (UID: \"92d984e8-b95b-4433-9a57-b5e15980a650\") " pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:01:56.073196 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.073171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret\") pod \"global-pull-secret-syncer-zf8vz\" (UID: \"0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4\") " pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:01:56.073438 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.073209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-slash\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.073438 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.073245    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d724302e-ab2b-46d7-ac99-0ebf76711d64-ovnkube-script-lib\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.073438 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.073277    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/441a3269-baa8-4575-99a6-007b21792801-hosts-file\") pod \"node-resolver-9b4sm\" (UID: \"441a3269-baa8-4575-99a6-007b21792801\") " pod="openshift-dns/node-resolver-9b4sm"
Jul 28 16:01:56.073438 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.073313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/201d9cf5-adcd-49dd-be6f-f8e20cedce09-konnectivity-ca\") pod \"konnectivity-agent-r4p2b\" (UID: \"201d9cf5-adcd-49dd-be6f-f8e20cedce09\") " pod="kube-system/konnectivity-agent-r4p2b"
Jul 28 16:01:56.073438 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.073390    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/441a3269-baa8-4575-99a6-007b21792801-hosts-file\") pod \"node-resolver-9b4sm\" (UID: \"441a3269-baa8-4575-99a6-007b21792801\") " pod="openshift-dns/node-resolver-9b4sm"
Jul 28 16:01:56.073670 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.073471    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-slash\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.073670 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:56.073541    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:56.074028 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.074007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-log-socket\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.074106 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.074054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-kubelet\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.074159 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.074105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.074159 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.074135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d724302e-ab2b-46d7-ac99-0ebf76711d64-ovnkube-config\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.074255 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.074168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/201d9cf5-adcd-49dd-be6f-f8e20cedce09-agent-certs\") pod \"konnectivity-agent-r4p2b\" (UID: \"201d9cf5-adcd-49dd-be6f-f8e20cedce09\") " pod="kube-system/konnectivity-agent-r4p2b"
Jul 28 16:01:56.074255 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.074201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvr7k\" (UniqueName: \"kubernetes.io/projected/441a3269-baa8-4575-99a6-007b21792801-kube-api-access-kvr7k\") pod \"node-resolver-9b4sm\" (UID: \"441a3269-baa8-4575-99a6-007b21792801\") " pod="openshift-dns/node-resolver-9b4sm"
Jul 28 16:01:56.074255 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.074235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-etc-openvswitch\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.074397 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.074266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-cni-bin\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.074397 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.074293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d724302e-ab2b-46d7-ac99-0ebf76711d64-env-overrides\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.074803 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.074776    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d724302e-ab2b-46d7-ac99-0ebf76711d64-env-overrides\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.074883 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:56.074870    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret podName:0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4 nodeName:}" failed. No retries permitted until 2026-07-28 16:01:56.574849561 +0000 UTC m=+3.133406089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret") pod "global-pull-secret-syncer-zf8vz" (UID: "0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:56.075653 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.075624    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/201d9cf5-adcd-49dd-be6f-f8e20cedce09-konnectivity-ca\") pod \"konnectivity-agent-r4p2b\" (UID: \"201d9cf5-adcd-49dd-be6f-f8e20cedce09\") " pod="kube-system/konnectivity-agent-r4p2b"
Jul 28 16:01:56.075921 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.075887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-log-socket\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.075991 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.075887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.075991 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.075962    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-cni-bin\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.076128 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.075918    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-host-kubelet\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.076189 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.076142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d724302e-ab2b-46d7-ac99-0ebf76711d64-ovnkube-script-lib\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.076189 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.076152    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d724302e-ab2b-46d7-ac99-0ebf76711d64-etc-openvswitch\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.077010 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.076982    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d724302e-ab2b-46d7-ac99-0ebf76711d64-ovnkube-config\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.077471 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.077444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d724302e-ab2b-46d7-ac99-0ebf76711d64-ovn-node-metrics-cert\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.078736 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.078718    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/201d9cf5-adcd-49dd-be6f-f8e20cedce09-agent-certs\") pod \"konnectivity-agent-r4p2b\" (UID: \"201d9cf5-adcd-49dd-be6f-f8e20cedce09\") " pod="kube-system/konnectivity-agent-r4p2b"
Jul 28 16:01:56.080673 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:56.080652    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:01:56.080762 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:56.080678    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:01:56.080762 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:56.080694    2581 projected.go:196] Error preparing data for projected volume kube-api-access-9p6kb for pod openshift-network-diagnostics/network-check-target-4hp9r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:56.080843 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:56.080761    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb podName:92d984e8-b95b-4433-9a57-b5e15980a650 nodeName:}" failed. No retries permitted until 2026-07-28 16:01:56.580742569 +0000 UTC m=+3.139299092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9p6kb" (UniqueName: "kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb") pod "network-check-target-4hp9r" (UID: "92d984e8-b95b-4433-9a57-b5e15980a650") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:56.082837 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.082814    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t5wz\" (UniqueName: \"kubernetes.io/projected/d724302e-ab2b-46d7-ac99-0ebf76711d64-kube-api-access-6t5wz\") pod \"ovnkube-node-khj85\" (UID: \"d724302e-ab2b-46d7-ac99-0ebf76711d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.086494 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.086434    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl72p\" (UniqueName: \"kubernetes.io/projected/4f2c74ea-8d67-407e-b213-adc356ab0a50-kube-api-access-vl72p\") pod \"iptables-alerter-nkcdf\" (UID: \"4f2c74ea-8d67-407e-b213-adc356ab0a50\") " pod="openshift-network-operator/iptables-alerter-nkcdf"
Jul 28 16:01:56.088009 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.087989    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvr7k\" (UniqueName: \"kubernetes.io/projected/441a3269-baa8-4575-99a6-007b21792801-kube-api-access-kvr7k\") pod \"node-resolver-9b4sm\" (UID: \"441a3269-baa8-4575-99a6-007b21792801\") " pod="openshift-dns/node-resolver-9b4sm"
Jul 28 16:01:56.158650 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.158617    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mrvh4"
Jul 28 16:01:56.167971 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.167944    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht"
Jul 28 16:01:56.181182 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.181164    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rwvsw"
Jul 28 16:01:56.185079 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.185059    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-44lcg"
Jul 28 16:01:56.192940 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.192921    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nnbz6"
Jul 28 16:01:56.201923 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.201904    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r4p2b"
Jul 28 16:01:56.210722 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.210702    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9b4sm"
Jul 28 16:01:56.219567 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.219542    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nkcdf"
Jul 28 16:01:56.226444 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.226422    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:01:56.477322 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.477289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs\") pod \"network-metrics-daemon-zf8jb\" (UID: \"c958899e-cac6-4e06-bf0a-e510c503c24d\") " pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:01:56.477487 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:56.477454    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:56.477546 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:56.477525    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs podName:c958899e-cac6-4e06-bf0a-e510c503c24d nodeName:}" failed. No retries permitted until 2026-07-28 16:01:57.477505985 +0000 UTC m=+4.036062507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs") pod "network-metrics-daemon-zf8jb" (UID: "c958899e-cac6-4e06-bf0a-e510c503c24d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:56.578697 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.578661    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret\") pod \"global-pull-secret-syncer-zf8vz\" (UID: \"0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4\") " pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:01:56.578888 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:56.578835    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:56.578957 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:56.578909    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret podName:0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4 nodeName:}" failed. No retries permitted until 2026-07-28 16:01:57.57889301 +0000 UTC m=+4.137449532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret") pod "global-pull-secret-syncer-zf8vz" (UID: "0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:56.679409 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.679378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p6kb\" (UniqueName: \"kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb\") pod \"network-check-target-4hp9r\" (UID: \"92d984e8-b95b-4433-9a57-b5e15980a650\") " pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:01:56.679563 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:56.679512    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:01:56.679563 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:56.679529    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:01:56.679563 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:56.679537    2581 projected.go:196] Error preparing data for projected volume kube-api-access-9p6kb for pod openshift-network-diagnostics/network-check-target-4hp9r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:56.679672 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:56.679590    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb podName:92d984e8-b95b-4433-9a57-b5e15980a650 nodeName:}" failed. No retries permitted until 2026-07-28 16:01:57.679574193 +0000 UTC m=+4.238130717 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9p6kb" (UniqueName: "kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb") pod "network-check-target-4hp9r" (UID: "92d984e8-b95b-4433-9a57-b5e15980a650") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:56.908870 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.908831    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 15:56:56 +0000 UTC" deadline="2027-12-29 15:59:03.593983817 +0000 UTC"
Jul 28 16:01:56.908870 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:56.908865    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12455h57m6.68512364s"
Jul 28 16:01:57.060208 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:57.060173    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d7a739_0ea4_4d89_8fd4_f302b556a891.slice/crio-5fc57283d2a034c28f1dc35081752f5b5d8aee74fe4185060ee6ef547c61e52f WatchSource:0}: Error finding container 5fc57283d2a034c28f1dc35081752f5b5d8aee74fe4185060ee6ef547c61e52f: Status 404 returned error can't find the container with id 5fc57283d2a034c28f1dc35081752f5b5d8aee74fe4185060ee6ef547c61e52f
Jul 28 16:01:57.063728 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:57.063669    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce86f1d2_93dd_4dde_9cda_f3e53fbd7642.slice/crio-e988454e6e6e5383aecdc795a287e775002c0c2567241c0bb4b469ba6f5f1ec2 WatchSource:0}: Error finding container e988454e6e6e5383aecdc795a287e775002c0c2567241c0bb4b469ba6f5f1ec2: Status 404 returned error can't find the container with id e988454e6e6e5383aecdc795a287e775002c0c2567241c0bb4b469ba6f5f1ec2
Jul 28 16:01:57.065535 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:57.065508    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd724302e_ab2b_46d7_ac99_0ebf76711d64.slice/crio-def38f4763510c1f823eaaf3c6d761be57cd18bf8781a7c6142fc5250755fb04 WatchSource:0}: Error finding container def38f4763510c1f823eaaf3c6d761be57cd18bf8781a7c6142fc5250755fb04: Status 404 returned error can't find the container with id def38f4763510c1f823eaaf3c6d761be57cd18bf8781a7c6142fc5250755fb04
Jul 28 16:01:57.066514 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:57.066492    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb400f70a_d50b_4eec_b1a4_fe911116339b.slice/crio-e9c4042504f38657d439651298f1a3c660156e43a2b670b7df241141a2eb0465 WatchSource:0}: Error finding container e9c4042504f38657d439651298f1a3c660156e43a2b670b7df241141a2eb0465: Status 404 returned error can't find the container with id e9c4042504f38657d439651298f1a3c660156e43a2b670b7df241141a2eb0465
Jul 28 16:01:57.067449 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:57.067378    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2c74ea_8d67_407e_b213_adc356ab0a50.slice/crio-c8cf4409e643118209a0c63071291cf6697dc02726f3c7f9800c5f9c1ed408a2 WatchSource:0}: Error finding container c8cf4409e643118209a0c63071291cf6697dc02726f3c7f9800c5f9c1ed408a2: Status 404 returned error can't find the container with id c8cf4409e643118209a0c63071291cf6697dc02726f3c7f9800c5f9c1ed408a2
Jul 28 16:01:57.068580 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:57.068380    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441a3269_baa8_4575_99a6_007b21792801.slice/crio-d1a2a7e6c5a9d1aeae1feb82dd175d7a9b82bb161bf5eb0ce1d9e90cdad8ae43 WatchSource:0}: Error finding container d1a2a7e6c5a9d1aeae1feb82dd175d7a9b82bb161bf5eb0ce1d9e90cdad8ae43: Status 404 returned error can't find the container with id d1a2a7e6c5a9d1aeae1feb82dd175d7a9b82bb161bf5eb0ce1d9e90cdad8ae43
Jul 28 16:01:57.069860 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:57.069406    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201d9cf5_adcd_49dd_be6f_f8e20cedce09.slice/crio-d35c1a2620eb2c2e58917ba008efc9ac3dfd16eca032bf5ef3018e8c8aa5e3f6 WatchSource:0}: Error finding container d35c1a2620eb2c2e58917ba008efc9ac3dfd16eca032bf5ef3018e8c8aa5e3f6: Status 404 returned error can't find the container with id d35c1a2620eb2c2e58917ba008efc9ac3dfd16eca032bf5ef3018e8c8aa5e3f6
Jul 28 16:01:57.070426 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:57.070208    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2e4031_a6e4_47a3_b57c_6767de81ffd9.slice/crio-9c89420165b2bb74bfcde7e120a3a9a785cbe2dec0769ae5ad5724e8909fd778 WatchSource:0}: Error finding container 9c89420165b2bb74bfcde7e120a3a9a785cbe2dec0769ae5ad5724e8909fd778: Status 404 returned error can't find the container with id 9c89420165b2bb74bfcde7e120a3a9a785cbe2dec0769ae5ad5724e8909fd778
Jul 28 16:01:57.071413 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:01:57.071313    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57f72c8_1133_4849_93f6_caa2d46580f4.slice/crio-fe2051c48208be4356650affe492948fec71e1250d3085faeb96e653c3814d1b WatchSource:0}: Error finding container fe2051c48208be4356650affe492948fec71e1250d3085faeb96e653c3814d1b: Status 404 returned error can't find the container with id fe2051c48208be4356650affe492948fec71e1250d3085faeb96e653c3814d1b
Jul 28 16:01:57.485701 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:57.485391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs\") pod \"network-metrics-daemon-zf8jb\" (UID: \"c958899e-cac6-4e06-bf0a-e510c503c24d\") " pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:01:57.485879 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:57.485706    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:57.485879 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:57.485788    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs podName:c958899e-cac6-4e06-bf0a-e510c503c24d nodeName:}" failed. No retries permitted until 2026-07-28 16:01:59.485770214 +0000 UTC m=+6.044326740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs") pod "network-metrics-daemon-zf8jb" (UID: "c958899e-cac6-4e06-bf0a-e510c503c24d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:57.586832 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:57.586710    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret\") pod \"global-pull-secret-syncer-zf8vz\" (UID: \"0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4\") " pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:01:57.586997 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:57.586848    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:57.586997 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:57.586945    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret podName:0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4 nodeName:}" failed. No retries permitted until 2026-07-28 16:01:59.586922993 +0000 UTC m=+6.145479517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret") pod "global-pull-secret-syncer-zf8vz" (UID: "0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:57.687140 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:57.687097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p6kb\" (UniqueName: \"kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb\") pod \"network-check-target-4hp9r\" (UID: \"92d984e8-b95b-4433-9a57-b5e15980a650\") " pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:01:57.687343 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:57.687326    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:01:57.687417 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:57.687352    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:01:57.687417 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:57.687366    2581 projected.go:196] Error preparing data for projected volume kube-api-access-9p6kb for pod openshift-network-diagnostics/network-check-target-4hp9r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:57.687527 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:57.687438    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb podName:92d984e8-b95b-4433-9a57-b5e15980a650 nodeName:}" failed. No retries permitted until 2026-07-28 16:01:59.687417143 +0000 UTC m=+6.245973687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9p6kb" (UniqueName: "kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb") pod "network-check-target-4hp9r" (UID: "92d984e8-b95b-4433-9a57-b5e15980a650") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:57.912153 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:57.909790    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 15:56:56 +0000 UTC" deadline="2028-03-23 04:06:31.601308165 +0000 UTC"
Jul 28 16:01:57.912153 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:57.909829    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14484h4m33.691484684s"
Jul 28 16:01:57.988535 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:57.988486    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:01:57.988712 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:57.988669    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hp9r" podUID="92d984e8-b95b-4433-9a57-b5e15980a650"
Jul 28 16:01:57.988808 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:57.988789    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:01:57.988941 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:57.988919    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf8vz" podUID="0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4"
Jul 28 16:01:57.989307 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:57.989285    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:01:57.989806 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:57.989778    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf8jb" podUID="c958899e-cac6-4e06-bf0a-e510c503c24d"
Jul 28 16:01:58.012238 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:58.012168    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nnbz6" event={"ID":"a57f72c8-1133-4849-93f6-caa2d46580f4","Type":"ContainerStarted","Data":"fe2051c48208be4356650affe492948fec71e1250d3085faeb96e653c3814d1b"}
Jul 28 16:01:58.026646 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:58.026615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r4p2b" event={"ID":"201d9cf5-adcd-49dd-be6f-f8e20cedce09","Type":"ContainerStarted","Data":"d35c1a2620eb2c2e58917ba008efc9ac3dfd16eca032bf5ef3018e8c8aa5e3f6"}
Jul 28 16:01:58.035994 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:58.035904    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht" event={"ID":"b400f70a-d50b-4eec-b1a4-fe911116339b","Type":"ContainerStarted","Data":"e9c4042504f38657d439651298f1a3c660156e43a2b670b7df241141a2eb0465"}
Jul 28 16:01:58.041485 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:58.041457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khj85" event={"ID":"d724302e-ab2b-46d7-ac99-0ebf76711d64","Type":"ContainerStarted","Data":"def38f4763510c1f823eaaf3c6d761be57cd18bf8781a7c6142fc5250755fb04"}
Jul 28 16:01:58.048255 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:58.048229    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-181.ec2.internal" event={"ID":"e97bc9874712512bf8be887ba1ad9d3d","Type":"ContainerStarted","Data":"11dbfde920d0e04277c09dbd0df03f96a2e74d1dd4f5144a7a0451f617cccddd"}
Jul 28 16:01:58.058878 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:58.058729    2581 generic.go:356] "Generic (PLEG): container finished" podID="bfce947cf24c76d49863c92e16d2c2ef" containerID="3e8a53bce6185b12daba755e9b99a03286e749e2a5bec2b67ab2efcf8cc2c926" exitCode=0
Jul 28 16:01:58.058878 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:58.058817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal" event={"ID":"bfce947cf24c76d49863c92e16d2c2ef","Type":"ContainerDied","Data":"3e8a53bce6185b12daba755e9b99a03286e749e2a5bec2b67ab2efcf8cc2c926"}
Jul 28 16:01:58.066790 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:58.065699    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-181.ec2.internal" podStartSLOduration=3.065684052 podStartE2EDuration="3.065684052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:01:58.065274956 +0000 UTC m=+4.623831523" watchObservedRunningTime="2026-07-28 16:01:58.065684052 +0000 UTC m=+4.624240598"
Jul 28 16:01:58.079959 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:58.079876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mrvh4" event={"ID":"9d2e4031-a6e4-47a3-b57c-6767de81ffd9","Type":"ContainerStarted","Data":"9c89420165b2bb74bfcde7e120a3a9a785cbe2dec0769ae5ad5724e8909fd778"}
Jul 28 16:01:58.094629 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:58.093309    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9b4sm" event={"ID":"441a3269-baa8-4575-99a6-007b21792801","Type":"ContainerStarted","Data":"d1a2a7e6c5a9d1aeae1feb82dd175d7a9b82bb161bf5eb0ce1d9e90cdad8ae43"}
Jul 28 16:01:58.098222 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:58.098198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nkcdf" event={"ID":"4f2c74ea-8d67-407e-b213-adc356ab0a50","Type":"ContainerStarted","Data":"c8cf4409e643118209a0c63071291cf6697dc02726f3c7f9800c5f9c1ed408a2"}
Jul 28 16:01:58.102286 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:58.102260    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-44lcg" event={"ID":"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642","Type":"ContainerStarted","Data":"e988454e6e6e5383aecdc795a287e775002c0c2567241c0bb4b469ba6f5f1ec2"}
Jul 28 16:01:58.107396 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:58.107369    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rwvsw" event={"ID":"f4d7a739-0ea4-4d89-8fd4-f302b556a891","Type":"ContainerStarted","Data":"5fc57283d2a034c28f1dc35081752f5b5d8aee74fe4185060ee6ef547c61e52f"}
Jul 28 16:01:59.143053 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:59.142855    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal" event={"ID":"bfce947cf24c76d49863c92e16d2c2ef","Type":"ContainerStarted","Data":"546ffd9f4b2be8caa9dc0ea54de1977a1c6cc4420f1ef6c172db2865340c8817"}
Jul 28 16:01:59.507776 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:59.507684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs\") pod \"network-metrics-daemon-zf8jb\" (UID: \"c958899e-cac6-4e06-bf0a-e510c503c24d\") " pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:01:59.507955 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:59.507844    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:59.507955 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:59.507908    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs podName:c958899e-cac6-4e06-bf0a-e510c503c24d nodeName:}" failed. No retries permitted until 2026-07-28 16:02:03.507887046 +0000 UTC m=+10.066443570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs") pod "network-metrics-daemon-zf8jb" (UID: "c958899e-cac6-4e06-bf0a-e510c503c24d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:59.609232 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:59.608567    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret\") pod \"global-pull-secret-syncer-zf8vz\" (UID: \"0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4\") " pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:01:59.609232 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:59.608777    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:59.609232 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:59.608841    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret podName:0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:03.608821237 +0000 UTC m=+10.167377774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret") pod "global-pull-secret-syncer-zf8vz" (UID: "0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:59.709150 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:59.709110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p6kb\" (UniqueName: \"kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb\") pod \"network-check-target-4hp9r\" (UID: \"92d984e8-b95b-4433-9a57-b5e15980a650\") " pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:01:59.709405 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:59.709348    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:01:59.709405 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:59.709374    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:01:59.709405 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:59.709388    2581 projected.go:196] Error preparing data for projected volume kube-api-access-9p6kb for pod openshift-network-diagnostics/network-check-target-4hp9r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:59.709626 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:59.709450    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb podName:92d984e8-b95b-4433-9a57-b5e15980a650 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:03.709429904 +0000 UTC m=+10.267986456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9p6kb" (UniqueName: "kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb") pod "network-check-target-4hp9r" (UID: "92d984e8-b95b-4433-9a57-b5e15980a650") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:59.980642 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:59.979849    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:01:59.980642 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:59.979991    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf8jb" podUID="c958899e-cac6-4e06-bf0a-e510c503c24d"
Jul 28 16:01:59.980642 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:59.980377    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:01:59.980642 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:59.980476    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hp9r" podUID="92d984e8-b95b-4433-9a57-b5e15980a650"
Jul 28 16:01:59.980642 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:01:59.980522    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:01:59.980642 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:01:59.980592    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf8vz" podUID="0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4"
Jul 28 16:02:01.978343 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:01.977733    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:01.978343 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:01.977762    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:01.978343 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:01.977741    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:01.978343 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:01.977891    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf8jb" podUID="c958899e-cac6-4e06-bf0a-e510c503c24d"
Jul 28 16:02:01.978343 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:01.977972    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hp9r" podUID="92d984e8-b95b-4433-9a57-b5e15980a650"
Jul 28 16:02:01.978343 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:01.978073    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf8vz" podUID="0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4"
Jul 28 16:02:03.545473 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:03.545208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs\") pod \"network-metrics-daemon-zf8jb\" (UID: \"c958899e-cac6-4e06-bf0a-e510c503c24d\") " pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:03.545473 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:03.545393    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:02:03.545473 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:03.545476    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs podName:c958899e-cac6-4e06-bf0a-e510c503c24d nodeName:}" failed. No retries permitted until 2026-07-28 16:02:11.545453483 +0000 UTC m=+18.104010007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs") pod "network-metrics-daemon-zf8jb" (UID: "c958899e-cac6-4e06-bf0a-e510c503c24d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:02:03.646018 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:03.645979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret\") pod \"global-pull-secret-syncer-zf8vz\" (UID: \"0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4\") " pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:03.646235 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:03.646159    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:02:03.646330 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:03.646310    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret podName:0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:11.646258811 +0000 UTC m=+18.204815352 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret") pod "global-pull-secret-syncer-zf8vz" (UID: "0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:02:03.747444 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:03.747400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p6kb\" (UniqueName: \"kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb\") pod \"network-check-target-4hp9r\" (UID: \"92d984e8-b95b-4433-9a57-b5e15980a650\") " pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:03.747654 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:03.747633    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:02:03.747743 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:03.747662    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:02:03.747743 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:03.747676    2581 projected.go:196] Error preparing data for projected volume kube-api-access-9p6kb for pod openshift-network-diagnostics/network-check-target-4hp9r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:02:03.747866 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:03.747748    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb podName:92d984e8-b95b-4433-9a57-b5e15980a650 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:11.747725896 +0000 UTC m=+18.306282427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9p6kb" (UniqueName: "kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb") pod "network-check-target-4hp9r" (UID: "92d984e8-b95b-4433-9a57-b5e15980a650") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:02:03.980575 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:03.979840    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:03.980575 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:03.979964    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hp9r" podUID="92d984e8-b95b-4433-9a57-b5e15980a650"
Jul 28 16:02:03.980575 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:03.980406    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:03.980575 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:03.980504    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf8vz" podUID="0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4"
Jul 28 16:02:03.981163 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:03.980990    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:03.981163 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:03.981116    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf8jb" podUID="c958899e-cac6-4e06-bf0a-e510c503c24d"
Jul 28 16:02:05.977876 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:05.977471    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:05.977876 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:05.977478    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:05.977876 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:05.977605    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf8vz" podUID="0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4"
Jul 28 16:02:05.977876 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:05.977481    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:05.977876 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:05.977716    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf8jb" podUID="c958899e-cac6-4e06-bf0a-e510c503c24d"
Jul 28 16:02:05.977876 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:05.977814    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hp9r" podUID="92d984e8-b95b-4433-9a57-b5e15980a650"
Jul 28 16:02:07.977830 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:07.977760    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:07.978298 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:07.977760    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:07.978298 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:07.977891    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf8jb" podUID="c958899e-cac6-4e06-bf0a-e510c503c24d"
Jul 28 16:02:07.978298 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:07.977956    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf8vz" podUID="0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4"
Jul 28 16:02:07.978298 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:07.977760    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:07.978298 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:07.978060    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hp9r" podUID="92d984e8-b95b-4433-9a57-b5e15980a650"
Jul 28 16:02:09.977747 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:09.977704    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:09.980996 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:09.977706    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:09.980996 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:09.977837    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf8vz" podUID="0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4"
Jul 28 16:02:09.980996 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:09.977727    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:09.980996 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:09.977908    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hp9r" podUID="92d984e8-b95b-4433-9a57-b5e15980a650"
Jul 28 16:02:09.980996 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:09.978051    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf8jb" podUID="c958899e-cac6-4e06-bf0a-e510c503c24d"
Jul 28 16:02:11.614676 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:11.614640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs\") pod \"network-metrics-daemon-zf8jb\" (UID: \"c958899e-cac6-4e06-bf0a-e510c503c24d\") " pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:11.615120 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:11.614748    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:02:11.615120 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:11.614798    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs podName:c958899e-cac6-4e06-bf0a-e510c503c24d nodeName:}" failed. No retries permitted until 2026-07-28 16:02:27.614784436 +0000 UTC m=+34.173340956 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs") pod "network-metrics-daemon-zf8jb" (UID: "c958899e-cac6-4e06-bf0a-e510c503c24d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:02:11.715706 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:11.715678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret\") pod \"global-pull-secret-syncer-zf8vz\" (UID: \"0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4\") " pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:11.715806 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:11.715770    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:02:11.715852 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:11.715813    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret podName:0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:27.71580013 +0000 UTC m=+34.274356650 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret") pod "global-pull-secret-syncer-zf8vz" (UID: "0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:02:11.816291 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:11.816264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p6kb\" (UniqueName: \"kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb\") pod \"network-check-target-4hp9r\" (UID: \"92d984e8-b95b-4433-9a57-b5e15980a650\") " pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:11.816418 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:11.816405    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:02:11.816453 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:11.816422    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:02:11.816453 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:11.816432    2581 projected.go:196] Error preparing data for projected volume kube-api-access-9p6kb for pod openshift-network-diagnostics/network-check-target-4hp9r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:02:11.816526 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:11.816471    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb podName:92d984e8-b95b-4433-9a57-b5e15980a650 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:27.816459203 +0000 UTC m=+34.375015726 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9p6kb" (UniqueName: "kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb") pod "network-check-target-4hp9r" (UID: "92d984e8-b95b-4433-9a57-b5e15980a650") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:02:11.977752 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:11.977707    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:11.977949 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:11.977707    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:11.977949 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:11.977837    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:11.977949 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:11.977836    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hp9r" podUID="92d984e8-b95b-4433-9a57-b5e15980a650"
Jul 28 16:02:11.978147 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:11.977956    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf8jb" podUID="c958899e-cac6-4e06-bf0a-e510c503c24d"
Jul 28 16:02:11.978147 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:11.978038    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf8vz" podUID="0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4"
Jul 28 16:02:13.978447 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:13.978409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:13.978896 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:13.978521    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:13.978896 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:13.978624    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:13.978896 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:13.978532    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf8jb" podUID="c958899e-cac6-4e06-bf0a-e510c503c24d"
Jul 28 16:02:13.978896 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:13.978698    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf8vz" podUID="0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4"
Jul 28 16:02:13.978896 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:13.978701    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hp9r" podUID="92d984e8-b95b-4433-9a57-b5e15980a650"
Jul 28 16:02:15.979387 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:15.978122    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:15.979387 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:15.978235    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hp9r" podUID="92d984e8-b95b-4433-9a57-b5e15980a650"
Jul 28 16:02:15.979387 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:15.978564    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:15.979387 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:15.978655    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf8vz" podUID="0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4"
Jul 28 16:02:15.979387 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:15.978804    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:15.979387 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:15.978894    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf8jb" podUID="c958899e-cac6-4e06-bf0a-e510c503c24d"
Jul 28 16:02:16.176311 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:16.175970    2581 generic.go:356] "Generic (PLEG): container finished" podID="a57f72c8-1133-4849-93f6-caa2d46580f4" containerID="7c404bf7260baf4210a95ee514e97f59f28fe2ec6589125387fe2e720774f4a5" exitCode=0
Jul 28 16:02:16.176311 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:16.176046    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nnbz6" event={"ID":"a57f72c8-1133-4849-93f6-caa2d46580f4","Type":"ContainerDied","Data":"7c404bf7260baf4210a95ee514e97f59f28fe2ec6589125387fe2e720774f4a5"}
Jul 28 16:02:16.177963 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:16.177938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r4p2b" event={"ID":"201d9cf5-adcd-49dd-be6f-f8e20cedce09","Type":"ContainerStarted","Data":"b85f2d25be4b18154a019142635af0252fc2b05b9e2eae69a58f55508542afc7"}
Jul 28 16:02:16.179407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:16.179386    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht" event={"ID":"b400f70a-d50b-4eec-b1a4-fe911116339b","Type":"ContainerStarted","Data":"64f8dc0db477146e37112c6b25eea3ff97ffa5c21fee38b9a4ce8e2c7c6fb3cf"}
Jul 28 16:02:16.184695 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:16.183208    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khj85" event={"ID":"d724302e-ab2b-46d7-ac99-0ebf76711d64","Type":"ContainerStarted","Data":"a4279ea9d31bfed75633bff4c1dc2e1fa0158a732aacb45a94561b75c14fed30"}
Jul 28 16:02:16.184695 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:16.183521    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khj85" event={"ID":"d724302e-ab2b-46d7-ac99-0ebf76711d64","Type":"ContainerStarted","Data":"fbb1f361062daa033b1eba85b6716f64a9934129afbdcf29147fad63cdfd89de"}
Jul 28 16:02:16.186173 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:16.186155    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mrvh4" event={"ID":"9d2e4031-a6e4-47a3-b57c-6767de81ffd9","Type":"ContainerStarted","Data":"51768e7d29f3ff095f96b8695ce1680d0ff860eca0e17a2a94d5808c3729b78f"}
Jul 28 16:02:16.187485 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:16.187461    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9b4sm" event={"ID":"441a3269-baa8-4575-99a6-007b21792801","Type":"ContainerStarted","Data":"18600b5eebc2dae3b4cf2c30f5dcf64ff242395ef5f7e718256dbd25e9f6cf63"}
Jul 28 16:02:16.188880 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:16.188859    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-44lcg" event={"ID":"ce86f1d2-93dd-4dde-9cda-f3e53fbd7642","Type":"ContainerStarted","Data":"d916c50ab825b13960b81c651764f6acc7cd30a95c8b540554c361b233d0440a"}
Jul 28 16:02:16.190461 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:16.190440    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rwvsw" event={"ID":"f4d7a739-0ea4-4d89-8fd4-f302b556a891","Type":"ContainerStarted","Data":"24c622999265fda9c39241d716c6f31297f77edd40484f66d57acf804ef17907"}
Jul 28 16:02:16.204704 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:16.204649    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-181.ec2.internal" podStartSLOduration=21.204637577 podStartE2EDuration="21.204637577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:01:59.163592702 +0000 UTC m=+5.722149247" watchObservedRunningTime="2026-07-28 16:02:16.204637577 +0000 UTC m=+22.763194119"
Jul 28 16:02:16.227492 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:16.227454    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mrvh4" podStartSLOduration=3.403660426 podStartE2EDuration="22.227439362s" totalImagesPullingTime="18.823778936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:16.227096392 +0000 UTC m=+22.785652933" watchObservedRunningTime="2026-07-28 16:02:16.227439362 +0000 UTC m=+22.785995904"
Jul 28 16:02:16.243496 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:16.243457    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rwvsw" podStartSLOduration=3.413279718 podStartE2EDuration="22.243442226s" totalImagesPullingTime="18.830162508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:16.243063641 +0000 UTC m=+22.801620194" watchObservedRunningTime="2026-07-28 16:02:16.243442226 +0000 UTC m=+22.801998821"
Jul 28 16:02:16.261946 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:16.261898    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-44lcg" podStartSLOduration=2.357236684 podStartE2EDuration="21.261878435s" totalImagesPullingTime="18.904641751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:16.261622506 +0000 UTC m=+22.820179042" watchObservedRunningTime="2026-07-28 16:02:16.261878435 +0000 UTC m=+22.820434981"
Jul 28 16:02:16.283922 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:16.283871    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r4p2b" podStartSLOduration=6.701254989 podStartE2EDuration="21.283849414s" totalImagesPullingTime="14.582594425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:16.283397362 +0000 UTC m=+22.841953905" watchObservedRunningTime="2026-07-28 16:02:16.283849414 +0000 UTC m=+22.842405958"
Jul 28 16:02:16.303383 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:16.303342    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9b4sm" podStartSLOduration=2.480979007 podStartE2EDuration="21.303327016s" totalImagesPullingTime="18.822348009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:16.302865329 +0000 UTC m=+22.861421853" watchObservedRunningTime="2026-07-28 16:02:16.303327016 +0000 UTC m=+22.861883558"
Jul 28 16:02:17.194728 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:17.194646    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nkcdf" event={"ID":"4f2c74ea-8d67-407e-b213-adc356ab0a50","Type":"ContainerStarted","Data":"d8739a71258e99187988df0b08c8ebda50bfc36a074dfe1ba4d966e0abfd2acc"}
Jul 28 16:02:17.198179 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:17.198087    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khj85" event={"ID":"d724302e-ab2b-46d7-ac99-0ebf76711d64","Type":"ContainerStarted","Data":"94af1ba42d2b8982206166cbe6e51129d9c01763447dc6b431276850fc633a02"}
Jul 28 16:02:17.198298 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:17.198186    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khj85" event={"ID":"d724302e-ab2b-46d7-ac99-0ebf76711d64","Type":"ContainerStarted","Data":"2d2d21a5058b62a7608599abbd2acdf5c0d40948dbd0b95cd11066752297c7eb"}
Jul 28 16:02:17.198298 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:17.198237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khj85" event={"ID":"d724302e-ab2b-46d7-ac99-0ebf76711d64","Type":"ContainerStarted","Data":"ba9e79dd3860d14b1dba8cd06fb85b1348acf28ea0a4d7146cef3b6aff0d9669"}
Jul 28 16:02:17.198415 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:17.198342    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khj85" event={"ID":"d724302e-ab2b-46d7-ac99-0ebf76711d64","Type":"ContainerStarted","Data":"001a2238d094000151568b1a661ea82243ecc835ea851e620d8cc2af0039eeec"}
Jul 28 16:02:17.212781 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:17.212741    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nkcdf" podStartSLOduration=3.387318787 podStartE2EDuration="22.212726227s" totalImagesPullingTime="18.82540744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:17.212241296 +0000 UTC m=+23.770797838" watchObservedRunningTime="2026-07-28 16:02:17.212726227 +0000 UTC m=+23.771282768"
Jul 28 16:02:17.731795 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:17.731762    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r4p2b"
Jul 28 16:02:17.732383 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:17.732361    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r4p2b"
Jul 28 16:02:17.977282 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:17.977248    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:17.977456 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:17.977248    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:17.977456 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:17.977378    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf8jb" podUID="c958899e-cac6-4e06-bf0a-e510c503c24d"
Jul 28 16:02:17.977572 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:17.977469    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hp9r" podUID="92d984e8-b95b-4433-9a57-b5e15980a650"
Jul 28 16:02:17.977572 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:17.977506    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:17.977660 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:17.977605    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf8vz" podUID="0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4"
Jul 28 16:02:19.205732 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:19.205711    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 16:02:19.205732 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:19.205718    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khj85" event={"ID":"d724302e-ab2b-46d7-ac99-0ebf76711d64","Type":"ContainerStarted","Data":"6c56896718dc7636fb47c5bad2e983039c8250ba3f9b50ce6ae3a829605e5039"}
Jul 28 16:02:19.977413 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:19.977383    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:19.977413 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:19.977408    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:19.977651 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:19.977384    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:19.977651 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:19.977527    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf8jb" podUID="c958899e-cac6-4e06-bf0a-e510c503c24d"
Jul 28 16:02:19.977760 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:19.977652    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hp9r" podUID="92d984e8-b95b-4433-9a57-b5e15980a650"
Jul 28 16:02:19.977760 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:19.977731    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf8vz" podUID="0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4"
Jul 28 16:02:21.111320 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:21.111297    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 16:02:21.212784 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:21.212622    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khj85" event={"ID":"d724302e-ab2b-46d7-ac99-0ebf76711d64","Type":"ContainerStarted","Data":"b7e30b631b0fb7024f5c31972591ae041dfc9a96f7e069037c64ac4e21ecd098"}
Jul 28 16:02:21.212923 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:21.212868    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:02:21.212923 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:21.212914    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:02:21.214490 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:21.214464    2581 generic.go:356] "Generic (PLEG): container finished" podID="a57f72c8-1133-4849-93f6-caa2d46580f4" containerID="9ea6eaaf0d27677fa3e4d9c78ecc1b5a237c8463efe5aaeeb3900d88f3ea00e4" exitCode=0
Jul 28 16:02:21.214584 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:21.214554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nnbz6" event={"ID":"a57f72c8-1133-4849-93f6-caa2d46580f4","Type":"ContainerDied","Data":"9ea6eaaf0d27677fa3e4d9c78ecc1b5a237c8463efe5aaeeb3900d88f3ea00e4"}
Jul 28 16:02:21.216442 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:21.216417    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht" event={"ID":"b400f70a-d50b-4eec-b1a4-fe911116339b","Type":"ContainerStarted","Data":"2c9cb5b0a3d84b0c61591d93f8035f94cfb265a3c3f944bda6ab280058979e2d"}
Jul 28 16:02:21.227901 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:21.227879    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:02:21.241627 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:21.241588    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-khj85" podStartSLOduration=7.352924743 podStartE2EDuration="26.241577182s" totalImagesPullingTime="18.888652439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:21.241122818 +0000 UTC m=+27.799679360" watchObservedRunningTime="2026-07-28 16:02:21.241577182 +0000 UTC m=+27.800133759"
Jul 28 16:02:21.927964 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:21.927666    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T16:02:21.111316098Z","UUID":"f3f0be94-dff8-4ba5-b112-d64a5002e417","Handler":null,"Name":"","Endpoint":""}
Jul 28 16:02:21.930729 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:21.930703    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 16:02:21.930862 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:21.930754    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 16:02:21.993037 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:21.981249    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:21.993037 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:21.981381    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf8jb" podUID="c958899e-cac6-4e06-bf0a-e510c503c24d"
Jul 28 16:02:21.993037 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:21.981733    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:21.993037 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:21.981820    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hp9r" podUID="92d984e8-b95b-4433-9a57-b5e15980a650"
Jul 28 16:02:21.993037 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:21.981868    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:21.993037 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:21.981928    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf8vz" podUID="0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4"
Jul 28 16:02:22.219360 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:22.219280    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:02:22.233970 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:22.233947    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:02:22.617851 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:22.617819    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r4p2b"
Jul 28 16:02:22.617962 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:22.617950    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 16:02:22.618327 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:22.618312    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r4p2b"
Jul 28 16:02:22.657191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:22.657160    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zf8jb"]
Jul 28 16:02:22.657308 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:22.657287    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:22.657407 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:22.657389    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf8jb" podUID="c958899e-cac6-4e06-bf0a-e510c503c24d"
Jul 28 16:02:22.665457 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:22.665413    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4hp9r"]
Jul 28 16:02:22.665586 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:22.665519    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:22.665649 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:22.665607    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hp9r" podUID="92d984e8-b95b-4433-9a57-b5e15980a650"
Jul 28 16:02:22.665970 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:22.665951    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zf8vz"]
Jul 28 16:02:22.666083 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:22.666069    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:22.666186 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:22.666167    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf8vz" podUID="0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4"
Jul 28 16:02:23.223499 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:23.223469    2581 generic.go:356] "Generic (PLEG): container finished" podID="a57f72c8-1133-4849-93f6-caa2d46580f4" containerID="3cb4181341f83b99b24557f35761595ccb9642bbdf79edcdab85065a0d2ffdaa" exitCode=0
Jul 28 16:02:23.223964 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:23.223550    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nnbz6" event={"ID":"a57f72c8-1133-4849-93f6-caa2d46580f4","Type":"ContainerDied","Data":"3cb4181341f83b99b24557f35761595ccb9642bbdf79edcdab85065a0d2ffdaa"}
Jul 28 16:02:23.225730 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:23.225705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht" event={"ID":"b400f70a-d50b-4eec-b1a4-fe911116339b","Type":"ContainerStarted","Data":"9fc13996bcc61237ec6202ac0d142d6184ddce82705a8db6e59dffd5b24ce1c2"}
Jul 28 16:02:23.978986 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:23.978955    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:23.979146 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:23.979097    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf8jb" podUID="c958899e-cac6-4e06-bf0a-e510c503c24d"
Jul 28 16:02:23.979205 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:23.979153    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:23.979287 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:23.979266    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hp9r" podUID="92d984e8-b95b-4433-9a57-b5e15980a650"
Jul 28 16:02:24.977564 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:24.977534    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:24.977862 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:24.977647    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf8vz" podUID="0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4"
Jul 28 16:02:25.232229 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:25.232040    2581 generic.go:356] "Generic (PLEG): container finished" podID="a57f72c8-1133-4849-93f6-caa2d46580f4" containerID="d834870a6a4580b0daf8c76dd686b5e56a6a3403a3d0a06561f7000211909010" exitCode=0
Jul 28 16:02:25.232229 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:25.232110    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nnbz6" event={"ID":"a57f72c8-1133-4849-93f6-caa2d46580f4","Type":"ContainerDied","Data":"d834870a6a4580b0daf8c76dd686b5e56a6a3403a3d0a06561f7000211909010"}
Jul 28 16:02:25.264129 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:25.264083    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxht" podStartSLOduration=6.152837988 podStartE2EDuration="31.264068191s" totalImagesPullingTime="25.111230203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:23.287386159 +0000 UTC m=+29.845942712" watchObservedRunningTime="2026-07-28 16:02:25.264068191 +0000 UTC m=+31.822624714"
Jul 28 16:02:25.977875 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:25.977843    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:25.978394 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:25.977984    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf8jb" podUID="c958899e-cac6-4e06-bf0a-e510c503c24d"
Jul 28 16:02:25.978394 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:25.978045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:25.978394 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:25.978161    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hp9r" podUID="92d984e8-b95b-4433-9a57-b5e15980a650"
Jul 28 16:02:26.977794 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:26.977761    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:26.977959 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:26.977881    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf8vz" podUID="0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4"
Jul 28 16:02:27.637957 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:27.637924    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs\") pod \"network-metrics-daemon-zf8jb\" (UID: \"c958899e-cac6-4e06-bf0a-e510c503c24d\") " pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:27.638251 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:27.638048    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:02:27.638251 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:27.638097    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs podName:c958899e-cac6-4e06-bf0a-e510c503c24d nodeName:}" failed. No retries permitted until 2026-07-28 16:02:59.638083863 +0000 UTC m=+66.196640383 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs") pod "network-metrics-daemon-zf8jb" (UID: "c958899e-cac6-4e06-bf0a-e510c503c24d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:02:27.739208 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:27.739176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret\") pod \"global-pull-secret-syncer-zf8vz\" (UID: \"0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4\") " pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:27.739334 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:27.739320    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:02:27.739393 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:27.739381    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret podName:0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:59.739364503 +0000 UTC m=+66.297921023 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret") pod "global-pull-secret-syncer-zf8vz" (UID: "0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:02:27.840381 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:27.840349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p6kb\" (UniqueName: \"kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb\") pod \"network-check-target-4hp9r\" (UID: \"92d984e8-b95b-4433-9a57-b5e15980a650\") " pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:27.840516 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:27.840503    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:02:27.840558 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:27.840529    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:02:27.840558 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:27.840539    2581 projected.go:196] Error preparing data for projected volume kube-api-access-9p6kb for pod openshift-network-diagnostics/network-check-target-4hp9r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:02:27.840641 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:27.840589    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb podName:92d984e8-b95b-4433-9a57-b5e15980a650 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:59.840570465 +0000 UTC m=+66.399126986 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9p6kb" (UniqueName: "kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb") pod "network-check-target-4hp9r" (UID: "92d984e8-b95b-4433-9a57-b5e15980a650") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:02:27.981221 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:27.981191    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:27.981727 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:27.981312    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hp9r" podUID="92d984e8-b95b-4433-9a57-b5e15980a650"
Jul 28 16:02:27.981727 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:27.981364    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:27.981727 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:27.981500    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf8jb" podUID="c958899e-cac6-4e06-bf0a-e510c503c24d"
Jul 28 16:02:28.771528 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.771436    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-181.ec2.internal" event="NodeReady"
Jul 28 16:02:28.771684 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.771586    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 16:02:28.819323 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.819296    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9c67998b-m5gg2"]
Jul 28 16:02:28.848230 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.848201    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"]
Jul 28 16:02:28.848380 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.848328    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9c67998b-m5gg2"
Jul 28 16:02:28.851368 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.851343    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-fndtg\""
Jul 28 16:02:28.853830 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.852183    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 16:02:28.853830 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.852395    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 16:02:28.853830 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.852997    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 16:02:28.857716 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.857676    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 16:02:28.869724 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.869696    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b487b6cd8-kjtj9"]
Jul 28 16:02:28.869836 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.869811    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:28.871917 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.871899    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 16:02:28.872053 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.872032    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 16:02:28.872263 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.872243    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 16:02:28.872349 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.872334    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 16:02:28.887597 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.887579    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp"]
Jul 28 16:02:28.887703 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.887685    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:28.890005 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.889965    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8qmb9\""
Jul 28 16:02:28.890114 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.890010    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 16:02:28.890114 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.889983    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 16:02:28.890287 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.889985    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 16:02:28.896901 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.896882    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 16:02:28.909736 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.909718    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9c67998b-m5gg2"]
Jul 28 16:02:28.909831 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.909745    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2pd4c"]
Jul 28 16:02:28.909831 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.909823    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp"
Jul 28 16:02:28.911967 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.911951    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 16:02:28.926922 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.926904    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp"]
Jul 28 16:02:28.927025 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.926940    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"]
Jul 28 16:02:28.927025 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.926958    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b487b6cd8-kjtj9"]
Jul 28 16:02:28.927025 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.926978    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v2tjl"]
Jul 28 16:02:28.927025 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.926993    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:02:28.928835 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.928815    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 16:02:28.928835 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.928831    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ssj24\""
Jul 28 16:02:28.929188 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.929167    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 16:02:28.948162 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-ca-trust-extracted\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:28.948258 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1b485cf-e0de-45fe-b32a-22220e0dccd9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c9c67998b-m5gg2\" (UID: \"e1b485cf-e0de-45fe-b32a-22220e0dccd9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9c67998b-m5gg2"
Jul 28 16:02:28.948318 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f85d2e6-2f7b-40d0-9128-9c46c323a42a-tmp\") pod \"klusterlet-addon-workmgr-8476b949c6-bx8dp\" (UID: \"6f85d2e6-2f7b-40d0-9128-9c46c323a42a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp"
Jul 28 16:02:28.948318 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-certificates\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:28.948435 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948325    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64de3fa9-b45e-4425-8706-990230e0dcc9-tmp-dir\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:02:28.948435 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948379    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77c3027-c745-4d5a-9157-8878d4e5be42-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8565d7f98f-zstd9\" (UID: \"e77c3027-c745-4d5a-9157-8878d4e5be42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:28.948435 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948411    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e77c3027-c745-4d5a-9157-8878d4e5be42-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8565d7f98f-zstd9\" (UID: \"e77c3027-c745-4d5a-9157-8878d4e5be42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:28.948582 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948488    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-trusted-ca\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:28.948582 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-installation-pull-secrets\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:28.948582 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948537    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:28.948582 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948560    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgvnx\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-kube-api-access-dgvnx\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:28.948780 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948602    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6f85d2e6-2f7b-40d0-9128-9c46c323a42a-klusterlet-config\") pod \"klusterlet-addon-workmgr-8476b949c6-bx8dp\" (UID: \"6f85d2e6-2f7b-40d0-9128-9c46c323a42a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp"
Jul 28 16:02:28.948780 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948627    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e77c3027-c745-4d5a-9157-8878d4e5be42-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8565d7f98f-zstd9\" (UID: \"e77c3027-c745-4d5a-9157-8878d4e5be42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:28.948780 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948652    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pplsq\" (UniqueName: \"kubernetes.io/projected/e77c3027-c745-4d5a-9157-8878d4e5be42-kube-api-access-pplsq\") pod \"cluster-proxy-proxy-agent-8565d7f98f-zstd9\" (UID: \"e77c3027-c745-4d5a-9157-8878d4e5be42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:28.948780 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948685    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-bound-sa-token\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:28.948780 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qnz8\" (UniqueName: \"kubernetes.io/projected/6f85d2e6-2f7b-40d0-9128-9c46c323a42a-kube-api-access-6qnz8\") pod \"klusterlet-addon-workmgr-8476b949c6-bx8dp\" (UID: \"6f85d2e6-2f7b-40d0-9128-9c46c323a42a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp"
Jul 28 16:02:28.948780 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948745    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e77c3027-c745-4d5a-9157-8878d4e5be42-ca\") pod \"cluster-proxy-proxy-agent-8565d7f98f-zstd9\" (UID: \"e77c3027-c745-4d5a-9157-8878d4e5be42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:28.948780 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-image-registry-private-configuration\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:28.949076 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948796    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwdd5\" (UniqueName: \"kubernetes.io/projected/64de3fa9-b45e-4425-8706-990230e0dcc9-kube-api-access-pwdd5\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:02:28.949076 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e77c3027-c745-4d5a-9157-8878d4e5be42-hub\") pod \"cluster-proxy-proxy-agent-8565d7f98f-zstd9\" (UID: \"e77c3027-c745-4d5a-9157-8878d4e5be42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:28.949076 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/64de3fa9-b45e-4425-8706-990230e0dcc9-config-volume\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:02:28.949076 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:02:28.949076 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.948892    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq96n\" (UniqueName: \"kubernetes.io/projected/e1b485cf-e0de-45fe-b32a-22220e0dccd9-kube-api-access-nq96n\") pod \"managed-serviceaccount-addon-agent-6c9c67998b-m5gg2\" (UID: \"e1b485cf-e0de-45fe-b32a-22220e0dccd9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9c67998b-m5gg2"
Jul 28 16:02:28.950968 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.950948    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2pd4c"]
Jul 28 16:02:28.951054 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.950986    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v2tjl"]
Jul 28 16:02:28.951097 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.951068    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v2tjl"
Jul 28 16:02:28.953485 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.953416    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 16:02:28.953485 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.953430    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 16:02:28.953485 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.953474    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bjngx\""
Jul 28 16:02:28.953698 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.953496    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 16:02:28.977143 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.977120    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:28.979348 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:28.979321    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 16:02:29.049456 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.049385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pplsq\" (UniqueName: \"kubernetes.io/projected/e77c3027-c745-4d5a-9157-8878d4e5be42-kube-api-access-pplsq\") pod \"cluster-proxy-proxy-agent-8565d7f98f-zstd9\" (UID: \"e77c3027-c745-4d5a-9157-8878d4e5be42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:29.049456 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.049425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-bound-sa-token\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:29.050034 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.049549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qnz8\" (UniqueName: \"kubernetes.io/projected/6f85d2e6-2f7b-40d0-9128-9c46c323a42a-kube-api-access-6qnz8\") pod \"klusterlet-addon-workmgr-8476b949c6-bx8dp\" (UID: \"6f85d2e6-2f7b-40d0-9128-9c46c323a42a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp"
Jul 28 16:02:29.050034 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.049581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e77c3027-c745-4d5a-9157-8878d4e5be42-ca\") pod \"cluster-proxy-proxy-agent-8565d7f98f-zstd9\" (UID: \"e77c3027-c745-4d5a-9157-8878d4e5be42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:29.050034 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.049616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-image-registry-private-configuration\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:29.050034 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.049637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwdd5\" (UniqueName: \"kubernetes.io/projected/64de3fa9-b45e-4425-8706-990230e0dcc9-kube-api-access-pwdd5\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:02:29.050034 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.049657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e77c3027-c745-4d5a-9157-8878d4e5be42-hub\") pod \"cluster-proxy-proxy-agent-8565d7f98f-zstd9\" (UID: \"e77c3027-c745-4d5a-9157-8878d4e5be42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:29.050034 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.049670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/64de3fa9-b45e-4425-8706-990230e0dcc9-config-volume\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:02:29.050034 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.049976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:02:29.050381 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.050048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq96n\" (UniqueName: \"kubernetes.io/projected/e1b485cf-e0de-45fe-b32a-22220e0dccd9-kube-api-access-nq96n\") pod \"managed-serviceaccount-addon-agent-6c9c67998b-m5gg2\" (UID: \"e1b485cf-e0de-45fe-b32a-22220e0dccd9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9c67998b-m5gg2"
Jul 28 16:02:29.058148 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.050082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-ca-trust-extracted\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:29.058249 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.058180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4f762d9d-a0ac-49c4-8920-1feb6ea59c89-cert\") pod \"ingress-canary-v2tjl\" (UID: \"4f762d9d-a0ac-49c4-8920-1feb6ea59c89\") " pod="openshift-ingress-canary/ingress-canary-v2tjl"
Jul 28 16:02:29.058249 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.058223    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44ncw\" (UniqueName: \"kubernetes.io/projected/4f762d9d-a0ac-49c4-8920-1feb6ea59c89-kube-api-access-44ncw\") pod \"ingress-canary-v2tjl\" (UID: \"4f762d9d-a0ac-49c4-8920-1feb6ea59c89\") " pod="openshift-ingress-canary/ingress-canary-v2tjl"
Jul 28 16:02:29.058362 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.058292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1b485cf-e0de-45fe-b32a-22220e0dccd9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c9c67998b-m5gg2\" (UID: \"e1b485cf-e0de-45fe-b32a-22220e0dccd9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9c67998b-m5gg2"
Jul 28 16:02:29.058362 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.058303    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/64de3fa9-b45e-4425-8706-990230e0dcc9-config-volume\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:02:29.058468 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.058373    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f85d2e6-2f7b-40d0-9128-9c46c323a42a-tmp\") pod \"klusterlet-addon-workmgr-8476b949c6-bx8dp\" (UID: \"6f85d2e6-2f7b-40d0-9128-9c46c323a42a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp"
Jul 28 16:02:29.058468 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.058418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-certificates\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:29.058468 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.058445    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64de3fa9-b45e-4425-8706-990230e0dcc9-tmp-dir\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:02:29.058617 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.058505    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77c3027-c745-4d5a-9157-8878d4e5be42-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8565d7f98f-zstd9\" (UID: \"e77c3027-c745-4d5a-9157-8878d4e5be42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:29.058617 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.058541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e77c3027-c745-4d5a-9157-8878d4e5be42-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8565d7f98f-zstd9\" (UID: \"e77c3027-c745-4d5a-9157-8878d4e5be42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:29.058617 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.058611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-trusted-ca\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:29.058767 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.058642    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-installation-pull-secrets\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:29.058767 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.058657    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-ca-trust-extracted\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:29.063095 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.063072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:29.063197 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.063120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvnx\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-kube-api-access-dgvnx\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:29.063197 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.063179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6f85d2e6-2f7b-40d0-9128-9c46c323a42a-klusterlet-config\") pod \"klusterlet-addon-workmgr-8476b949c6-bx8dp\" (UID: \"6f85d2e6-2f7b-40d0-9128-9c46c323a42a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp"
Jul 28 16:02:29.063306 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.063214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e77c3027-c745-4d5a-9157-8878d4e5be42-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8565d7f98f-zstd9\" (UID: \"e77c3027-c745-4d5a-9157-8878d4e5be42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:29.064334 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.064308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64de3fa9-b45e-4425-8706-990230e0dcc9-tmp-dir\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:02:29.064417 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.064390    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-certificates\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:29.064816 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.064791    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f85d2e6-2f7b-40d0-9128-9c46c323a42a-tmp\") pod \"klusterlet-addon-workmgr-8476b949c6-bx8dp\" (UID: \"6f85d2e6-2f7b-40d0-9128-9c46c323a42a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp"
Jul 28 16:02:29.064912 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:29.064897    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:02:29.065103 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.064952    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-trusted-ca\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:29.065103 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:29.064985    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls podName:64de3fa9-b45e-4425-8706-990230e0dcc9 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:29.564954956 +0000 UTC m=+36.123511480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls") pod "dns-default-2pd4c" (UID: "64de3fa9-b45e-4425-8706-990230e0dcc9") : secret "dns-default-metrics-tls" not found
Jul 28 16:02:29.065103 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.065030    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77c3027-c745-4d5a-9157-8878d4e5be42-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8565d7f98f-zstd9\" (UID: \"e77c3027-c745-4d5a-9157-8878d4e5be42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:29.065475 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.065259    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwdd5\" (UniqueName: \"kubernetes.io/projected/64de3fa9-b45e-4425-8706-990230e0dcc9-kube-api-access-pwdd5\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:02:29.065574 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:29.065530    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 16:02:29.065574 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:29.065545    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b487b6cd8-kjtj9: secret "image-registry-tls" not found
Jul 28 16:02:29.067069 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:29.065607    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls podName:1a151931-b3f7-4dd6-8cf9-10b6d57d62e3 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:29.565591103 +0000 UTC m=+36.124147627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls") pod "image-registry-b487b6cd8-kjtj9" (UID: "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3") : secret "image-registry-tls" not found
Jul 28 16:02:29.067069 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.065824    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r4p2b_201d9cf5-adcd-49dd-be6f-f8e20cedce09/konnectivity-agent/0.log" limit=1024
Jul 28 16:02:29.067069 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.066785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pplsq\" (UniqueName: \"kubernetes.io/projected/e77c3027-c745-4d5a-9157-8878d4e5be42-kube-api-access-pplsq\") pod \"cluster-proxy-proxy-agent-8565d7f98f-zstd9\" (UID: \"e77c3027-c745-4d5a-9157-8878d4e5be42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:29.067979 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.067828    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6f85d2e6-2f7b-40d0-9128-9c46c323a42a-klusterlet-config\") pod \"klusterlet-addon-workmgr-8476b949c6-bx8dp\" (UID: \"6f85d2e6-2f7b-40d0-9128-9c46c323a42a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp"
Jul 28 16:02:29.068693 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.068614    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1b485cf-e0de-45fe-b32a-22220e0dccd9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c9c67998b-m5gg2\" (UID: \"e1b485cf-e0de-45fe-b32a-22220e0dccd9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9c67998b-m5gg2"
Jul 28 16:02:29.068693 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.068676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-image-registry-private-configuration\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:29.069115 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.069072    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e77c3027-c745-4d5a-9157-8878d4e5be42-hub\") pod \"cluster-proxy-proxy-agent-8565d7f98f-zstd9\" (UID: \"e77c3027-c745-4d5a-9157-8878d4e5be42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:29.069219 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.068648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e77c3027-c745-4d5a-9157-8878d4e5be42-ca\") pod \"cluster-proxy-proxy-agent-8565d7f98f-zstd9\" (UID: \"e77c3027-c745-4d5a-9157-8878d4e5be42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:29.069940 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.069917    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-installation-pull-secrets\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:29.070221 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.070195    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e77c3027-c745-4d5a-9157-8878d4e5be42-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8565d7f98f-zstd9\" (UID: \"e77c3027-c745-4d5a-9157-8878d4e5be42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:29.070312 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.070260    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e77c3027-c745-4d5a-9157-8878d4e5be42-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8565d7f98f-zstd9\" (UID: \"e77c3027-c745-4d5a-9157-8878d4e5be42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:29.070927 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.070906    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-bound-sa-token\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:29.071270 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.071251    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qnz8\" (UniqueName: \"kubernetes.io/projected/6f85d2e6-2f7b-40d0-9128-9c46c323a42a-kube-api-access-6qnz8\") pod \"klusterlet-addon-workmgr-8476b949c6-bx8dp\" (UID: \"6f85d2e6-2f7b-40d0-9128-9c46c323a42a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp"
Jul 28 16:02:29.078403 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.078384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq96n\" (UniqueName: \"kubernetes.io/projected/e1b485cf-e0de-45fe-b32a-22220e0dccd9-kube-api-access-nq96n\") pod \"managed-serviceaccount-addon-agent-6c9c67998b-m5gg2\" (UID: \"e1b485cf-e0de-45fe-b32a-22220e0dccd9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9c67998b-m5gg2"
Jul 28 16:02:29.079481 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.079459    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvnx\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-kube-api-access-dgvnx\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:29.160738 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.160697    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r4p2b_201d9cf5-adcd-49dd-be6f-f8e20cedce09/konnectivity-agent/0.log" limit=1024
Jul 28 16:02:29.163667 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.163645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4f762d9d-a0ac-49c4-8920-1feb6ea59c89-cert\") pod \"ingress-canary-v2tjl\" (UID: \"4f762d9d-a0ac-49c4-8920-1feb6ea59c89\") " pod="openshift-ingress-canary/ingress-canary-v2tjl"
Jul 28 16:02:29.163800 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.163676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44ncw\" (UniqueName: \"kubernetes.io/projected/4f762d9d-a0ac-49c4-8920-1feb6ea59c89-kube-api-access-44ncw\") pod \"ingress-canary-v2tjl\" (UID: \"4f762d9d-a0ac-49c4-8920-1feb6ea59c89\") " pod="openshift-ingress-canary/ingress-canary-v2tjl"
Jul 28 16:02:29.166036 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.166002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4f762d9d-a0ac-49c4-8920-1feb6ea59c89-cert\") pod \"ingress-canary-v2tjl\" (UID: \"4f762d9d-a0ac-49c4-8920-1feb6ea59c89\") " pod="openshift-ingress-canary/ingress-canary-v2tjl"
Jul 28 16:02:29.172090 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.172042    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9c67998b-m5gg2"
Jul 28 16:02:29.174038 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.174000    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44ncw\" (UniqueName: \"kubernetes.io/projected/4f762d9d-a0ac-49c4-8920-1feb6ea59c89-kube-api-access-44ncw\") pod \"ingress-canary-v2tjl\" (UID: \"4f762d9d-a0ac-49c4-8920-1feb6ea59c89\") " pod="openshift-ingress-canary/ingress-canary-v2tjl"
Jul 28 16:02:29.180804 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.180761    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:02:29.219610 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.219586    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp"
Jul 28 16:02:29.260683 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.260664    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v2tjl"
Jul 28 16:02:29.471120 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.470891    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"]
Jul 28 16:02:29.476870 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.476847    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9c67998b-m5gg2"]
Jul 28 16:02:29.477907 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.477717    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v2tjl"]
Jul 28 16:02:29.488112 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.488095    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp"]
Jul 28 16:02:29.522892 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:02:29.522869    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode77c3027_c745_4d5a_9157_8878d4e5be42.slice/crio-659f49c55e4d2c5b847f348517e78d03ff6d24d19ed2849dabd9ae1b8efc631d WatchSource:0}: Error finding container 659f49c55e4d2c5b847f348517e78d03ff6d24d19ed2849dabd9ae1b8efc631d: Status 404 returned error can't find the container with id 659f49c55e4d2c5b847f348517e78d03ff6d24d19ed2849dabd9ae1b8efc631d
Jul 28 16:02:29.523370 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:02:29.523346    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f762d9d_a0ac_49c4_8920_1feb6ea59c89.slice/crio-a1dc5f659d17768035788799e4c7af3c56b47b0df2ef09e98b3fda34c1db81c4 WatchSource:0}: Error finding container a1dc5f659d17768035788799e4c7af3c56b47b0df2ef09e98b3fda34c1db81c4: Status 404 returned error can't find the container with id a1dc5f659d17768035788799e4c7af3c56b47b0df2ef09e98b3fda34c1db81c4
Jul 28 16:02:29.524152 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:02:29.524129    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1b485cf_e0de_45fe_b32a_22220e0dccd9.slice/crio-1695f20a510986511f3c9a60c2a0e867aa1a73ab85eec3c581f2a7ac415edab3 WatchSource:0}: Error finding container 1695f20a510986511f3c9a60c2a0e867aa1a73ab85eec3c581f2a7ac415edab3: Status 404 returned error can't find the container with id 1695f20a510986511f3c9a60c2a0e867aa1a73ab85eec3c581f2a7ac415edab3
Jul 28 16:02:29.524851 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:02:29.524829    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f85d2e6_2f7b_40d0_9128_9c46c323a42a.slice/crio-b210c7a1412fab399099fe06549091b66863bd682f12d309b6223948e23a0aae WatchSource:0}: Error finding container b210c7a1412fab399099fe06549091b66863bd682f12d309b6223948e23a0aae: Status 404 returned error can't find the container with id b210c7a1412fab399099fe06549091b66863bd682f12d309b6223948e23a0aae
Jul 28 16:02:29.567126 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.567105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:29.567202 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.567149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:02:29.567254 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:29.567243    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:02:29.567291 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:29.567284    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls podName:64de3fa9-b45e-4425-8706-990230e0dcc9 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:30.56727226 +0000 UTC m=+37.125828780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls") pod "dns-default-2pd4c" (UID: "64de3fa9-b45e-4425-8706-990230e0dcc9") : secret "dns-default-metrics-tls" not found
Jul 28 16:02:29.567291 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:29.567243    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 16:02:29.567357 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:29.567299    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b487b6cd8-kjtj9: secret "image-registry-tls" not found
Jul 28 16:02:29.567357 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:29.567318    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls podName:1a151931-b3f7-4dd6-8cf9-10b6d57d62e3 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:30.567312076 +0000 UTC m=+37.125868596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls") pod "image-registry-b487b6cd8-kjtj9" (UID: "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3") : secret "image-registry-tls" not found
Jul 28 16:02:29.977663 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.977628    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:29.978062 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.977827    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:29.980966 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.980930    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bvgkt\""
Jul 28 16:02:29.981111 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.980976    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f7pxp\""
Jul 28 16:02:29.981111 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.980976    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 16:02:29.981111 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.981042    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 16:02:29.981111 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:29.980937    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 16:02:30.245127 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:30.245051    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9" event={"ID":"e77c3027-c745-4d5a-9157-8878d4e5be42","Type":"ContainerStarted","Data":"659f49c55e4d2c5b847f348517e78d03ff6d24d19ed2849dabd9ae1b8efc631d"}
Jul 28 16:02:30.249728 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:30.249702    2581 generic.go:356] "Generic (PLEG): container finished" podID="a57f72c8-1133-4849-93f6-caa2d46580f4" containerID="149af0700c3435111025e1104fc12db36b47c0d2f0095857ec237c9ed7a85b0f" exitCode=0
Jul 28 16:02:30.249852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:30.249795    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nnbz6" event={"ID":"a57f72c8-1133-4849-93f6-caa2d46580f4","Type":"ContainerDied","Data":"149af0700c3435111025e1104fc12db36b47c0d2f0095857ec237c9ed7a85b0f"}
Jul 28 16:02:30.251415 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:30.251389    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v2tjl" event={"ID":"4f762d9d-a0ac-49c4-8920-1feb6ea59c89","Type":"ContainerStarted","Data":"a1dc5f659d17768035788799e4c7af3c56b47b0df2ef09e98b3fda34c1db81c4"}
Jul 28 16:02:30.253277 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:30.253251    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp" event={"ID":"6f85d2e6-2f7b-40d0-9128-9c46c323a42a","Type":"ContainerStarted","Data":"b210c7a1412fab399099fe06549091b66863bd682f12d309b6223948e23a0aae"}
Jul 28 16:02:30.254808 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:30.254786    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9c67998b-m5gg2" event={"ID":"e1b485cf-e0de-45fe-b32a-22220e0dccd9","Type":"ContainerStarted","Data":"1695f20a510986511f3c9a60c2a0e867aa1a73ab85eec3c581f2a7ac415edab3"}
Jul 28 16:02:30.575687 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:30.575615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:02:30.575821 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:30.575711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:30.575886 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:30.575859    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 16:02:30.575886 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:30.575875    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b487b6cd8-kjtj9: secret "image-registry-tls" not found
Jul 28 16:02:30.575975 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:30.575936    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls podName:1a151931-b3f7-4dd6-8cf9-10b6d57d62e3 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:32.575915963 +0000 UTC m=+39.134472502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls") pod "image-registry-b487b6cd8-kjtj9" (UID: "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3") : secret "image-registry-tls" not found
Jul 28 16:02:30.576465 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:30.576347    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:02:30.576465 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:30.576402    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls podName:64de3fa9-b45e-4425-8706-990230e0dcc9 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:32.576386021 +0000 UTC m=+39.134942559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls") pod "dns-default-2pd4c" (UID: "64de3fa9-b45e-4425-8706-990230e0dcc9") : secret "dns-default-metrics-tls" not found
Jul 28 16:02:31.264246 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:31.264106    2581 generic.go:356] "Generic (PLEG): container finished" podID="a57f72c8-1133-4849-93f6-caa2d46580f4" containerID="7d666e09e85829cc967be973e0bf4dd77ac17448ba3c1133b1d252d4a7ba59a6" exitCode=0
Jul 28 16:02:31.264246 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:31.264185    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nnbz6" event={"ID":"a57f72c8-1133-4849-93f6-caa2d46580f4","Type":"ContainerDied","Data":"7d666e09e85829cc967be973e0bf4dd77ac17448ba3c1133b1d252d4a7ba59a6"}
Jul 28 16:02:32.273212 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:32.272948    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nnbz6" event={"ID":"a57f72c8-1133-4849-93f6-caa2d46580f4","Type":"ContainerStarted","Data":"35cbceccb0fa0065ad8d7a77f43650d2417e6c073bf632f9a98e1261633f9355"}
Jul 28 16:02:32.304472 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:32.304240    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nnbz6" podStartSLOduration=6.949634883 podStartE2EDuration="37.304219498s" totalImagesPullingTime="30.354584615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:32.303762522 +0000 UTC m=+38.862319065" watchObservedRunningTime="2026-07-28 16:02:32.304219498 +0000 UTC m=+38.862776057"
Jul 28 16:02:32.439484 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:32.439430    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r4p2b_201d9cf5-adcd-49dd-be6f-f8e20cedce09/konnectivity-agent/0.log" limit=1024
Jul 28 16:02:32.540325 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:32.540243    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r4p2b_201d9cf5-adcd-49dd-be6f-f8e20cedce09/konnectivity-agent/0.log" limit=1024
Jul 28 16:02:32.596367 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:32.596219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:02:32.596367 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:32.596323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:32.596367 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:32.596358    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:02:32.596367 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:32.596430    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls podName:64de3fa9-b45e-4425-8706-990230e0dcc9 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:36.59640786 +0000 UTC m=+43.154964393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls") pod "dns-default-2pd4c" (UID: "64de3fa9-b45e-4425-8706-990230e0dcc9") : secret "dns-default-metrics-tls" not found
Jul 28 16:02:32.596786 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:32.596450    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 16:02:32.596786 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:32.596468    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b487b6cd8-kjtj9: secret "image-registry-tls" not found
Jul 28 16:02:32.596786 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:32.596522    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls podName:1a151931-b3f7-4dd6-8cf9-10b6d57d62e3 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:36.596502659 +0000 UTC m=+43.155059201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls") pod "image-registry-b487b6cd8-kjtj9" (UID: "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3") : secret "image-registry-tls" not found
Jul 28 16:02:34.123492 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:34.123445    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r4p2b_201d9cf5-adcd-49dd-be6f-f8e20cedce09/konnectivity-agent/0.log" limit=1024
Jul 28 16:02:34.228123 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:34.228068    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r4p2b_201d9cf5-adcd-49dd-be6f-f8e20cedce09/konnectivity-agent/0.log" limit=1024
Jul 28 16:02:36.629562 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:36.629522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:36.629958 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:36.629585    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:02:36.629958 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:36.629679    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:02:36.629958 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:36.629698    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 16:02:36.629958 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:36.629721    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b487b6cd8-kjtj9: secret "image-registry-tls" not found
Jul 28 16:02:36.629958 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:36.629731    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls podName:64de3fa9-b45e-4425-8706-990230e0dcc9 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:44.629716274 +0000 UTC m=+51.188272794 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls") pod "dns-default-2pd4c" (UID: "64de3fa9-b45e-4425-8706-990230e0dcc9") : secret "dns-default-metrics-tls" not found
Jul 28 16:02:36.629958 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:36.629788    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls podName:1a151931-b3f7-4dd6-8cf9-10b6d57d62e3 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:44.629769293 +0000 UTC m=+51.188325826 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls") pod "image-registry-b487b6cd8-kjtj9" (UID: "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3") : secret "image-registry-tls" not found
Jul 28 16:02:38.294458 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:38.294420    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9c67998b-m5gg2" event={"ID":"e1b485cf-e0de-45fe-b32a-22220e0dccd9","Type":"ContainerStarted","Data":"1222d7a9cb9335825f9f211a808b874c19c761a1101dca17d05bcd0a97514e06"}
Jul 28 16:02:38.296597 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:38.296566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9" event={"ID":"e77c3027-c745-4d5a-9157-8878d4e5be42","Type":"ContainerStarted","Data":"49517509e08784ca2e84fdf34b83d8c7923604707d29c545ee256d8abafaa894"}
Jul 28 16:02:38.296738 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:38.296607    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9" event={"ID":"e77c3027-c745-4d5a-9157-8878d4e5be42","Type":"ContainerStarted","Data":"6835f2712822a5fd49b5c4cb8ba6038ece8b4f588c909f4732a9d2c02a1bc5db"}
Jul 28 16:02:38.296738 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:38.296620    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9" event={"ID":"e77c3027-c745-4d5a-9157-8878d4e5be42","Type":"ContainerStarted","Data":"0a0659be64b10440f9414e578cf125fadf3ccb9033309ef86413b84841528bb9"}
Jul 28 16:02:38.297873 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:38.297847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v2tjl" event={"ID":"4f762d9d-a0ac-49c4-8920-1feb6ea59c89","Type":"ContainerStarted","Data":"4aec9d559904d69f849ed7a52064c6c02717ad24758e2337d900cdb676793275"}
Jul 28 16:02:38.299280 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:38.299254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp" event={"ID":"6f85d2e6-2f7b-40d0-9128-9c46c323a42a","Type":"ContainerStarted","Data":"1c95fb6e27f1c0be6fd36b7e74117fdcf1787c7045961802e6d0611af803406f"}
Jul 28 16:02:38.299457 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:38.299441    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp"
Jul 28 16:02:38.301083 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:38.301065    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp"
Jul 28 16:02:38.315672 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:38.315636    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9c67998b-m5gg2" podStartSLOduration=28.573201068 podStartE2EDuration="36.315609672s" totalImagesPullingTime="7.742408604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:38.315062459 +0000 UTC m=+44.873619011" watchObservedRunningTime="2026-07-28 16:02:38.315609672 +0000 UTC m=+44.874166207"
Jul 28 16:02:38.335094 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:38.335047    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9" podStartSLOduration=28.57729133 podStartE2EDuration="36.335032715s" totalImagesPullingTime="7.757741385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:38.334352565 +0000 UTC m=+44.892909118" watchObservedRunningTime="2026-07-28 16:02:38.335032715 +0000 UTC m=+44.893589247"
Jul 28 16:02:38.357380 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:38.357333    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8476b949c6-bx8dp" podStartSLOduration=28.614660154 podStartE2EDuration="36.357319677s" totalImagesPullingTime="7.742659523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:38.35570874 +0000 UTC m=+44.914265280" watchObservedRunningTime="2026-07-28 16:02:38.357319677 +0000 UTC m=+44.915876222"
Jul 28 16:02:38.381172 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:38.381126    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v2tjl" podStartSLOduration=2.637921884 podStartE2EDuration="10.381111573s" totalImagesPullingTime="7.743189689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:38.380917575 +0000 UTC m=+44.939474127" watchObservedRunningTime="2026-07-28 16:02:38.381111573 +0000 UTC m=+44.939668114"
Jul 28 16:02:44.690364 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:44.690328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:02:44.690846 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:44.690422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:02:44.690846 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:44.690483    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:02:44.690846 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:44.690551    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 16:02:44.690846 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:44.690562    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls podName:64de3fa9-b45e-4425-8706-990230e0dcc9 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:00.69054472 +0000 UTC m=+67.249101241 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls") pod "dns-default-2pd4c" (UID: "64de3fa9-b45e-4425-8706-990230e0dcc9") : secret "dns-default-metrics-tls" not found
Jul 28 16:02:44.690846 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:44.690566    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b487b6cd8-kjtj9: secret "image-registry-tls" not found
Jul 28 16:02:44.690846 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:44.690620    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls podName:1a151931-b3f7-4dd6-8cf9-10b6d57d62e3 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:00.690603741 +0000 UTC m=+67.249160287 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls") pod "image-registry-b487b6cd8-kjtj9" (UID: "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3") : secret "image-registry-tls" not found
Jul 28 16:02:54.238185 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:54.238157    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khj85"
Jul 28 16:02:59.698150 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:59.698112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs\") pod \"network-metrics-daemon-zf8jb\" (UID: \"c958899e-cac6-4e06-bf0a-e510c503c24d\") " pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:02:59.700652 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:59.700631    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 16:02:59.708364 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:59.708346    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 16:02:59.708464 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:02:59.708438    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs podName:c958899e-cac6-4e06-bf0a-e510c503c24d nodeName:}" failed. No retries permitted until 2026-07-28 16:04:03.708416546 +0000 UTC m=+130.266973074 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs") pod "network-metrics-daemon-zf8jb" (UID: "c958899e-cac6-4e06-bf0a-e510c503c24d") : secret "metrics-daemon-secret" not found
Jul 28 16:02:59.798491 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:59.798465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret\") pod \"global-pull-secret-syncer-zf8vz\" (UID: \"0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4\") " pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:59.800569 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:59.800554    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 16:02:59.812547 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:59.812524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4-original-pull-secret\") pod \"global-pull-secret-syncer-zf8vz\" (UID: \"0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4\") " pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:59.886426 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:59.886403    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf8vz"
Jul 28 16:02:59.899700 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:59.899679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p6kb\" (UniqueName: \"kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb\") pod \"network-check-target-4hp9r\" (UID: \"92d984e8-b95b-4433-9a57-b5e15980a650\") " pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:59.902155 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:59.902134    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 16:02:59.912793 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:59.912771    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 16:02:59.923608 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:59.923592    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p6kb\" (UniqueName: \"kubernetes.io/projected/92d984e8-b95b-4433-9a57-b5e15980a650-kube-api-access-9p6kb\") pod \"network-check-target-4hp9r\" (UID: \"92d984e8-b95b-4433-9a57-b5e15980a650\") " pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:02:59.993140 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:02:59.993091    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bvgkt\""
Jul 28 16:03:00.001059 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:00.001037    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:03:00.038266 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:00.038240    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zf8vz"]
Jul 28 16:03:00.041898 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:03:00.041864    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f92d8f5_b089_4ccf_a2c2_524a0eb11ef4.slice/crio-bf7743ff510cf287af1580afdb0517d2066ede8945e58325014dfd96df6b6511 WatchSource:0}: Error finding container bf7743ff510cf287af1580afdb0517d2066ede8945e58325014dfd96df6b6511: Status 404 returned error can't find the container with id bf7743ff510cf287af1580afdb0517d2066ede8945e58325014dfd96df6b6511
Jul 28 16:03:00.142385 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:00.142351    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4hp9r"]
Jul 28 16:03:00.145402 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:03:00.145372    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d984e8_b95b_4433_9a57_b5e15980a650.slice/crio-664abc830d8347b76898f7fa8671914094630a342e1bfc5b7f1af918cc659f7f WatchSource:0}: Error finding container 664abc830d8347b76898f7fa8671914094630a342e1bfc5b7f1af918cc659f7f: Status 404 returned error can't find the container with id 664abc830d8347b76898f7fa8671914094630a342e1bfc5b7f1af918cc659f7f
Jul 28 16:03:00.360589 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:00.360531    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4hp9r" event={"ID":"92d984e8-b95b-4433-9a57-b5e15980a650","Type":"ContainerStarted","Data":"664abc830d8347b76898f7fa8671914094630a342e1bfc5b7f1af918cc659f7f"}
Jul 28 16:03:00.361527 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:00.361505    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zf8vz" event={"ID":"0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4","Type":"ContainerStarted","Data":"bf7743ff510cf287af1580afdb0517d2066ede8945e58325014dfd96df6b6511"}
Jul 28 16:03:00.705871 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:00.705847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:03:00.706188 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:00.705892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:03:00.706188 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:00.706010    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:03:00.706188 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:00.706032    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 16:03:00.706188 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:00.706043    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b487b6cd8-kjtj9: secret "image-registry-tls" not found
Jul 28 16:03:00.706188 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:00.706085    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls podName:64de3fa9-b45e-4425-8706-990230e0dcc9 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:32.70606738 +0000 UTC m=+99.264623907 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls") pod "dns-default-2pd4c" (UID: "64de3fa9-b45e-4425-8706-990230e0dcc9") : secret "dns-default-metrics-tls" not found
Jul 28 16:03:00.706188 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:00.706100    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls podName:1a151931-b3f7-4dd6-8cf9-10b6d57d62e3 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:32.706093896 +0000 UTC m=+99.264650417 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls") pod "image-registry-b487b6cd8-kjtj9" (UID: "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3") : secret "image-registry-tls" not found
Jul 28 16:03:07.386394 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:07.386289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4hp9r" event={"ID":"92d984e8-b95b-4433-9a57-b5e15980a650","Type":"ContainerStarted","Data":"6296b7ca404c76ea83bea256e59dbc8a7cb53c68693054d42de49c5222641815"}
Jul 28 16:03:07.386899 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:07.386418    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:03:07.387797 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:07.387763    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zf8vz" event={"ID":"0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4","Type":"ContainerStarted","Data":"a85f0b738a898d3d4c92e90b726befc622a9380e4c6c65815d2a0ea595ab9b11"}
Jul 28 16:03:07.405342 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:07.405302    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4hp9r" podStartSLOduration=67.241638434 podStartE2EDuration="1m12.405289159s" totalImagesPullingTime="5.163650725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:03:07.404710569 +0000 UTC m=+73.963267093" watchObservedRunningTime="2026-07-28 16:03:07.405289159 +0000 UTC m=+73.963845701"
Jul 28 16:03:07.422239 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:07.422199    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zf8vz" podStartSLOduration=65.329904158 podStartE2EDuration="1m12.422188017s" totalImagesPullingTime="7.092283859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:03:07.421581074 +0000 UTC m=+73.980137617" watchObservedRunningTime="2026-07-28 16:03:07.422188017 +0000 UTC m=+73.980744850"
Jul 28 16:03:27.513476 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.513362    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rq7b2"]
Jul 28 16:03:27.517549 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.517533    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:27.519719 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.519695    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nbttq\""
Jul 28 16:03:27.519824 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.519705    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 16:03:27.520119 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.520100    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 16:03:27.520207 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.520109    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 16:03:27.520207 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.520129    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 16:03:27.539692 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.539668    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rq7b2"]
Jul 28 16:03:27.691478 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.691444    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4lnj\" (UniqueName: \"kubernetes.io/projected/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-api-access-q4lnj\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:27.691478 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.691480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-crio-socket\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:27.691697 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.691501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:27.691697 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.691545    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-data-volume\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:27.691697 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.691579    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:27.792917 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.792810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-crio-socket\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:27.792917 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.792866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:27.793151 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:27.792938    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 16:03:27.793151 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.792957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-data-volume\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:27.793151 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.792982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:27.793151 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:27.793041    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-rbac-proxy-cm podName:2a9f054b-0e6c-4f0b-a83e-235ce83c1a58 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:28.292995464 +0000 UTC m=+94.851551998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rq7b2" (UID: "2a9f054b-0e6c-4f0b-a83e-235ce83c1a58") : configmap "kube-rbac-proxy" not found
Jul 28 16:03:27.793151 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.792938    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-crio-socket\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:27.793151 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:27.793112    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:27.793366 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:27.793162    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls podName:2a9f054b-0e6c-4f0b-a83e-235ce83c1a58 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:28.293143606 +0000 UTC m=+94.851700141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rq7b2" (UID: "2a9f054b-0e6c-4f0b-a83e-235ce83c1a58") : secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:27.793366 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.793231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4lnj\" (UniqueName: \"kubernetes.io/projected/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-api-access-q4lnj\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:27.793366 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.793265    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-data-volume\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:27.803880 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:27.803854    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4lnj\" (UniqueName: \"kubernetes.io/projected/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-api-access-q4lnj\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:28.297086 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:28.297055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:28.297254 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:28.297096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:28.297254 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:28.297158    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 16:03:28.297254 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:28.297204    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:28.297254 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:28.297221    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-rbac-proxy-cm podName:2a9f054b-0e6c-4f0b-a83e-235ce83c1a58 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:29.297206599 +0000 UTC m=+95.855763119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rq7b2" (UID: "2a9f054b-0e6c-4f0b-a83e-235ce83c1a58") : configmap "kube-rbac-proxy" not found
Jul 28 16:03:28.297254 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:28.297240    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls podName:2a9f054b-0e6c-4f0b-a83e-235ce83c1a58 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:29.297229324 +0000 UTC m=+95.855785848 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rq7b2" (UID: "2a9f054b-0e6c-4f0b-a83e-235ce83c1a58") : secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:29.304266 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:29.304207    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:29.304266 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:29.304269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:29.304770 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:29.304351    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 16:03:29.304770 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:29.304419    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:29.304770 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:29.304427    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-rbac-proxy-cm podName:2a9f054b-0e6c-4f0b-a83e-235ce83c1a58 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:31.30441155 +0000 UTC m=+97.862968094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rq7b2" (UID: "2a9f054b-0e6c-4f0b-a83e-235ce83c1a58") : configmap "kube-rbac-proxy" not found
Jul 28 16:03:29.304770 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:29.304491    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls podName:2a9f054b-0e6c-4f0b-a83e-235ce83c1a58 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:31.304473834 +0000 UTC m=+97.863030359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rq7b2" (UID: "2a9f054b-0e6c-4f0b-a83e-235ce83c1a58") : secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:29.311684 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:29.311664    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9b4sm_441a3269-baa8-4575-99a6-007b21792801/dns-node-resolver/0.log"
Jul 28 16:03:30.512145 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:30.512115    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rwvsw_f4d7a739-0ea4-4d89-8fd4-f302b556a891/node-ca/0.log"
Jul 28 16:03:31.315072 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:31.315044    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v2tjl_4f762d9d-a0ac-49c4-8920-1feb6ea59c89/serve-healthcheck-canary/0.log"
Jul 28 16:03:31.320224 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:31.320204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:31.320284 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:31.320235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:31.320327 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:31.320313    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 16:03:31.320370 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:31.320359    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:31.320404 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:31.320369    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-rbac-proxy-cm podName:2a9f054b-0e6c-4f0b-a83e-235ce83c1a58 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:35.320354205 +0000 UTC m=+101.878910725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rq7b2" (UID: "2a9f054b-0e6c-4f0b-a83e-235ce83c1a58") : configmap "kube-rbac-proxy" not found
Jul 28 16:03:31.320444 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:31.320409    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls podName:2a9f054b-0e6c-4f0b-a83e-235ce83c1a58 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:35.320398228 +0000 UTC m=+101.878954752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rq7b2" (UID: "2a9f054b-0e6c-4f0b-a83e-235ce83c1a58") : secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:32.731669 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:32.731630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:03:32.731669 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:32.731672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls\") pod \"image-registry-b487b6cd8-kjtj9\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") " pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:03:32.732114 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:32.731766    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 16:03:32.732114 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:32.731779    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b487b6cd8-kjtj9: secret "image-registry-tls" not found
Jul 28 16:03:32.732114 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:32.731763    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:03:32.732114 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:32.731837    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls podName:1a151931-b3f7-4dd6-8cf9-10b6d57d62e3 nodeName:}" failed. No retries permitted until 2026-07-28 16:04:36.731823398 +0000 UTC m=+163.290379931 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls") pod "image-registry-b487b6cd8-kjtj9" (UID: "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3") : secret "image-registry-tls" not found
Jul 28 16:03:32.732114 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:32.731850    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls podName:64de3fa9-b45e-4425-8706-990230e0dcc9 nodeName:}" failed. No retries permitted until 2026-07-28 16:04:36.731844928 +0000 UTC m=+163.290401448 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls") pod "dns-default-2pd4c" (UID: "64de3fa9-b45e-4425-8706-990230e0dcc9") : secret "dns-default-metrics-tls" not found
Jul 28 16:03:35.349584 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:35.349532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:35.349584 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:35.349584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:35.350144 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:35.349664    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 16:03:35.350144 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:35.349680    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:35.350144 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:35.349730    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls podName:2a9f054b-0e6c-4f0b-a83e-235ce83c1a58 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:43.349718323 +0000 UTC m=+109.908274842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rq7b2" (UID: "2a9f054b-0e6c-4f0b-a83e-235ce83c1a58") : secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:35.350144 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:35.349768    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-rbac-proxy-cm podName:2a9f054b-0e6c-4f0b-a83e-235ce83c1a58 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:43.349741467 +0000 UTC m=+109.908297986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rq7b2" (UID: "2a9f054b-0e6c-4f0b-a83e-235ce83c1a58") : configmap "kube-rbac-proxy" not found
Jul 28 16:03:38.393297 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:38.393269    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4hp9r"
Jul 28 16:03:43.408471 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:43.408421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:43.408471 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:43.408474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:43.408909 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:43.408601    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:43.408909 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:03:43.408663    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls podName:2a9f054b-0e6c-4f0b-a83e-235ce83c1a58 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:59.408647918 +0000 UTC m=+125.967204438 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rq7b2" (UID: "2a9f054b-0e6c-4f0b-a83e-235ce83c1a58") : secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:43.408982 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:43.408939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:49.181996 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:49.181950    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9" podUID="e77c3027-c745-4d5a-9157-8878d4e5be42" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 16:03:59.182430 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:59.182380    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9" podUID="e77c3027-c745-4d5a-9157-8878d4e5be42" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 16:03:59.432305 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:59.432261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:59.434620 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:59.434565    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a9f054b-0e6c-4f0b-a83e-235ce83c1a58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq7b2\" (UID: \"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58\") " pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:59.628113 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:59.628082    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nbttq\""
Jul 28 16:03:59.636847 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:59.636822    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rq7b2"
Jul 28 16:03:59.770382 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:03:59.770352    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rq7b2"]
Jul 28 16:03:59.773583 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:03:59.773555    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9f054b_0e6c_4f0b_a83e_235ce83c1a58.slice/crio-ec172a060ac820c250d4af55bd28102553797532a52f5f8ff802de4e8c95ae57 WatchSource:0}: Error finding container ec172a060ac820c250d4af55bd28102553797532a52f5f8ff802de4e8c95ae57: Status 404 returned error can't find the container with id ec172a060ac820c250d4af55bd28102553797532a52f5f8ff802de4e8c95ae57
Jul 28 16:04:00.536523 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:00.536486    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rq7b2" event={"ID":"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58","Type":"ContainerStarted","Data":"ec172a060ac820c250d4af55bd28102553797532a52f5f8ff802de4e8c95ae57"}
Jul 28 16:04:03.769714 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:03.769675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs\") pod \"network-metrics-daemon-zf8jb\" (UID: \"c958899e-cac6-4e06-bf0a-e510c503c24d\") " pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:04:03.772342 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:03.772316    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c958899e-cac6-4e06-bf0a-e510c503c24d-metrics-certs\") pod \"network-metrics-daemon-zf8jb\" (UID: \"c958899e-cac6-4e06-bf0a-e510c503c24d\") " pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:04:03.899932 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:03.899899    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f7pxp\""
Jul 28 16:04:03.908139 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:03.908122    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf8jb"
Jul 28 16:04:04.773464 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:04.773438    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zf8jb"]
Jul 28 16:04:04.782692 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:04:04.782659    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc958899e_cac6_4e06_bf0a_e510c503c24d.slice/crio-14b4e6f786799d9c079b3c5c5a788f0027d450f8b8d10b27c0ce18d242bef522 WatchSource:0}: Error finding container 14b4e6f786799d9c079b3c5c5a788f0027d450f8b8d10b27c0ce18d242bef522: Status 404 returned error can't find the container with id 14b4e6f786799d9c079b3c5c5a788f0027d450f8b8d10b27c0ce18d242bef522
Jul 28 16:04:05.555835 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:05.555777    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rq7b2" event={"ID":"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58","Type":"ContainerStarted","Data":"1b5eda2dfbcd475a13ec58ba3383b9f2e92723207303c2be447a0c180511fcb9"}
Jul 28 16:04:05.557278 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:05.557232    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zf8jb" event={"ID":"c958899e-cac6-4e06-bf0a-e510c503c24d","Type":"ContainerStarted","Data":"14b4e6f786799d9c079b3c5c5a788f0027d450f8b8d10b27c0ce18d242bef522"}
Jul 28 16:04:06.562053 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:06.562000    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zf8jb" event={"ID":"c958899e-cac6-4e06-bf0a-e510c503c24d","Type":"ContainerStarted","Data":"24cb809c01c9d0014e01771e075e84523b0d3b96d51928215b9cd477a9d00705"}
Jul 28 16:04:06.562440 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:06.562082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zf8jb" event={"ID":"c958899e-cac6-4e06-bf0a-e510c503c24d","Type":"ContainerStarted","Data":"4b9fe291d4a5d6c363c10797a025363f98fa9ea501f4ded08a66d2f49e18592d"}
Jul 28 16:04:06.590949 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:06.590895    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zf8jb" podStartSLOduration=131.540024096 podStartE2EDuration="2m12.590875431s" totalImagesPullingTime="1.050851335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:06.589766221 +0000 UTC m=+133.148322766" watchObservedRunningTime="2026-07-28 16:04:06.590875431 +0000 UTC m=+133.149431974"
Jul 28 16:04:08.570628 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:08.570539    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rq7b2" event={"ID":"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58","Type":"ContainerStarted","Data":"18c7322bdbadf55eb42abd342b5b6f631c40db3d4efc469f26ecc3d302a9bd47"}
Jul 28 16:04:09.182355 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:09.182310    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9" podUID="e77c3027-c745-4d5a-9157-8878d4e5be42" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 16:04:09.182521 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:09.182396    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9"
Jul 28 16:04:09.182959 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:09.182912    2581 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"49517509e08784ca2e84fdf34b83d8c7923604707d29c545ee256d8abafaa894"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 28 16:04:09.183002 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:09.182986    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9" podUID="e77c3027-c745-4d5a-9157-8878d4e5be42" containerName="service-proxy" containerID="cri-o://49517509e08784ca2e84fdf34b83d8c7923604707d29c545ee256d8abafaa894" gracePeriod=30
Jul 28 16:04:09.576705 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:09.576622    2581 generic.go:356] "Generic (PLEG): container finished" podID="e77c3027-c745-4d5a-9157-8878d4e5be42" containerID="49517509e08784ca2e84fdf34b83d8c7923604707d29c545ee256d8abafaa894" exitCode=2
Jul 28 16:04:09.577212 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:09.576700    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9" event={"ID":"e77c3027-c745-4d5a-9157-8878d4e5be42","Type":"ContainerDied","Data":"49517509e08784ca2e84fdf34b83d8c7923604707d29c545ee256d8abafaa894"}
Jul 28 16:04:09.577212 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:09.576759    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8565d7f98f-zstd9" event={"ID":"e77c3027-c745-4d5a-9157-8878d4e5be42","Type":"ContainerStarted","Data":"43f4ceffabb08331cf3d5ea2ee3a30e9615606a2f2177952c3005f96655794d3"}
Jul 28 16:04:12.589284 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:12.589248    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rq7b2" event={"ID":"2a9f054b-0e6c-4f0b-a83e-235ce83c1a58","Type":"ContainerStarted","Data":"3be72dd396aee503cfa886f3387b95460d7aac5dcaa6e7418ac6d4d4bacd7ec0"}
Jul 28 16:04:12.612334 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:12.612288    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rq7b2" podStartSLOduration=33.776507854 podStartE2EDuration="45.612275408s" totalImagesPullingTime="11.835767554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:03:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:12.611947856 +0000 UTC m=+139.170504397" watchObservedRunningTime="2026-07-28 16:04:12.612275408 +0000 UTC m=+139.170831949"
Jul 28 16:04:18.730764 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.730732    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5cvfq"]
Jul 28 16:04:18.734006 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.733986    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.735719 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.735701    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5dx87\""
Jul 28 16:04:18.736209 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.736184    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 16:04:18.736666 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.736646    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 16:04:18.736666 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.736666    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 16:04:18.736666 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.736670    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 16:04:18.736877 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.736675    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 16:04:18.738172 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.738158    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 16:04:18.887943 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.887913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc3defc1-78da-434a-a933-cba987e0d5c0-sys\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.887943 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.887942    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc3defc1-78da-434a-a933-cba987e0d5c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.888131 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.887985    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc3defc1-78da-434a-a933-cba987e0d5c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.888131 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.888031    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc3defc1-78da-434a-a933-cba987e0d5c0-node-exporter-tls\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.888131 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.888087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc3defc1-78da-434a-a933-cba987e0d5c0-node-exporter-textfile\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.888131 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.888115    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc3defc1-78da-434a-a933-cba987e0d5c0-metrics-client-ca\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.888307 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.888133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc3defc1-78da-434a-a933-cba987e0d5c0-root\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.888307 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.888152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc3defc1-78da-434a-a933-cba987e0d5c0-node-exporter-wtmp\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.888307 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.888198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqkkr\" (UniqueName: \"kubernetes.io/projected/dc3defc1-78da-434a-a933-cba987e0d5c0-kube-api-access-wqkkr\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.988572 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.988509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc3defc1-78da-434a-a933-cba987e0d5c0-node-exporter-textfile\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.988572 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.988541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc3defc1-78da-434a-a933-cba987e0d5c0-metrics-client-ca\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.988572 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.988567    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc3defc1-78da-434a-a933-cba987e0d5c0-root\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.988796 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.988593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc3defc1-78da-434a-a933-cba987e0d5c0-node-exporter-wtmp\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.988796 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.988623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqkkr\" (UniqueName: \"kubernetes.io/projected/dc3defc1-78da-434a-a933-cba987e0d5c0-kube-api-access-wqkkr\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.988796 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.988656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc3defc1-78da-434a-a933-cba987e0d5c0-sys\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.988796 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.988675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc3defc1-78da-434a-a933-cba987e0d5c0-root\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.988796 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.988679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc3defc1-78da-434a-a933-cba987e0d5c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.988796 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.988741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc3defc1-78da-434a-a933-cba987e0d5c0-sys\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.989128 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.988798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc3defc1-78da-434a-a933-cba987e0d5c0-node-exporter-wtmp\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.989128 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.988814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc3defc1-78da-434a-a933-cba987e0d5c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.989128 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.988841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc3defc1-78da-434a-a933-cba987e0d5c0-node-exporter-tls\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.989128 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.988852    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc3defc1-78da-434a-a933-cba987e0d5c0-node-exporter-textfile\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.989334 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.989195    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc3defc1-78da-434a-a933-cba987e0d5c0-metrics-client-ca\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.989384 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.989350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc3defc1-78da-434a-a933-cba987e0d5c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.990953 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.990933    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc3defc1-78da-434a-a933-cba987e0d5c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.991057 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.991042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc3defc1-78da-434a-a933-cba987e0d5c0-node-exporter-tls\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:18.999953 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:18.999927    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqkkr\" (UniqueName: \"kubernetes.io/projected/dc3defc1-78da-434a-a933-cba987e0d5c0-kube-api-access-wqkkr\") pod \"node-exporter-5cvfq\" (UID: \"dc3defc1-78da-434a-a933-cba987e0d5c0\") " pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:19.043234 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:19.043209    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5cvfq"
Jul 28 16:04:19.053090 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:04:19.053072    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3defc1_78da_434a_a933_cba987e0d5c0.slice/crio-5962389af01778956447dd8bb387dd030dfcf2309dcc8ebe03effc55a0228215 WatchSource:0}: Error finding container 5962389af01778956447dd8bb387dd030dfcf2309dcc8ebe03effc55a0228215: Status 404 returned error can't find the container with id 5962389af01778956447dd8bb387dd030dfcf2309dcc8ebe03effc55a0228215
Jul 28 16:04:19.612116 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:19.612082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5cvfq" event={"ID":"dc3defc1-78da-434a-a933-cba987e0d5c0","Type":"ContainerStarted","Data":"5962389af01778956447dd8bb387dd030dfcf2309dcc8ebe03effc55a0228215"}
Jul 28 16:04:20.617359 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:20.617328    2581 generic.go:356] "Generic (PLEG): container finished" podID="dc3defc1-78da-434a-a933-cba987e0d5c0" containerID="5e038861b0dc4155a6327fb841ebd87d3109591f640b37666cd41759fe7bcdc7" exitCode=0
Jul 28 16:04:20.617801 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:20.617371    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5cvfq" event={"ID":"dc3defc1-78da-434a-a933-cba987e0d5c0","Type":"ContainerDied","Data":"5e038861b0dc4155a6327fb841ebd87d3109591f640b37666cd41759fe7bcdc7"}
Jul 28 16:04:21.624399 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:21.624356    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5cvfq" event={"ID":"dc3defc1-78da-434a-a933-cba987e0d5c0","Type":"ContainerStarted","Data":"3a8956d3e5a0c65503bd02fea51e6a8d32e2a693dc63609cb3ae1cb65aca3b1c"}
Jul 28 16:04:21.624873 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:21.624415    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5cvfq" event={"ID":"dc3defc1-78da-434a-a933-cba987e0d5c0","Type":"ContainerStarted","Data":"f4d159b943663b86a2c561339e6b2fd6a9ae2edfffb2399b8a108c70a432c2fb"}
Jul 28 16:04:21.646842 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:21.646795    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5cvfq" podStartSLOduration=2.738769971 podStartE2EDuration="3.646779792s" totalImagesPullingTime="846.40345ms" totalInitContainerRuntime="61.606371ms" isStatefulPod=true podCreationTimestamp="2026-07-28 16:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:21.645512864 +0000 UTC m=+148.204069406" watchObservedRunningTime="2026-07-28 16:04:21.646779792 +0000 UTC m=+148.205336334"
Jul 28 16:04:28.895365 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:28.895333    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b487b6cd8-kjtj9"]
Jul 28 16:04:28.895740 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:04:28.895547    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9" podUID="1a151931-b3f7-4dd6-8cf9-10b6d57d62e3"
Jul 28 16:04:29.649800 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.649766    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:04:29.656839 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.656816    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:04:29.770045 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.770003    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-installation-pull-secrets\") pod \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") "
Jul 28 16:04:29.770045 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.770053    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-trusted-ca\") pod \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") "
Jul 28 16:04:29.770262 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.770084    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-certificates\") pod \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") "
Jul 28 16:04:29.770262 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.770116    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-bound-sa-token\") pod \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") "
Jul 28 16:04:29.770262 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.770145    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-ca-trust-extracted\") pod \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") "
Jul 28 16:04:29.770262 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.770171    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgvnx\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-kube-api-access-dgvnx\") pod \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") "
Jul 28 16:04:29.770262 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.770210    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-image-registry-private-configuration\") pod \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\" (UID: \"1a151931-b3f7-4dd6-8cf9-10b6d57d62e3\") "
Jul 28 16:04:29.770494 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.770457    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3" (UID: "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:04:29.770550 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.770486    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3" (UID: "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:04:29.770607 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.770550    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3" (UID: "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:04:29.772439 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.772404    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3" (UID: "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:04:29.772554 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.772457    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3" (UID: "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:04:29.772554 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.772463    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-kube-api-access-dgvnx" (OuterVolumeSpecName: "kube-api-access-dgvnx") pod "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3" (UID: "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3"). InnerVolumeSpecName "kube-api-access-dgvnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:04:29.772554 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.772510    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3" (UID: "1a151931-b3f7-4dd6-8cf9-10b6d57d62e3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:04:29.871616 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.871587    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-installation-pull-secrets\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:04:29.871616 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.871612    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-trusted-ca\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:04:29.871616 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.871622    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-certificates\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:04:29.871803 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.871630    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-bound-sa-token\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:04:29.871803 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.871639    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-ca-trust-extracted\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:04:29.871803 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.871647    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgvnx\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-kube-api-access-dgvnx\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:04:29.871803 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:29.871656    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-image-registry-private-configuration\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:04:30.653279 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:30.653246    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b487b6cd8-kjtj9"
Jul 28 16:04:30.686485 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:30.686455    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b487b6cd8-kjtj9"]
Jul 28 16:04:30.691200 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:30.691179    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b487b6cd8-kjtj9"]
Jul 28 16:04:30.779522 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:30.779497    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3-registry-tls\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:04:31.936263 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:04:31.936223    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-2pd4c" podUID="64de3fa9-b45e-4425-8706-990230e0dcc9"
Jul 28 16:04:31.981372 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:31.981336    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a151931-b3f7-4dd6-8cf9-10b6d57d62e3" path="/var/lib/kubelet/pods/1a151931-b3f7-4dd6-8cf9-10b6d57d62e3/volumes"
Jul 28 16:04:32.659030 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:32.658990    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:04:36.819731 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:36.819688    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:04:36.821967 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:36.821948    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64de3fa9-b45e-4425-8706-990230e0dcc9-metrics-tls\") pod \"dns-default-2pd4c\" (UID: \"64de3fa9-b45e-4425-8706-990230e0dcc9\") " pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:04:36.862668 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:36.862642    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ssj24\""
Jul 28 16:04:36.870199 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:36.870179    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:04:37.108833 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:37.108755    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2pd4c"]
Jul 28 16:04:37.111961 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:04:37.111933    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64de3fa9_b45e_4425_8706_990230e0dcc9.slice/crio-0c2aaff65713cd1034a4f01c67463c450e84e89ce3fe48b29217e24f371f241b WatchSource:0}: Error finding container 0c2aaff65713cd1034a4f01c67463c450e84e89ce3fe48b29217e24f371f241b: Status 404 returned error can't find the container with id 0c2aaff65713cd1034a4f01c67463c450e84e89ce3fe48b29217e24f371f241b
Jul 28 16:04:37.674824 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:37.674794    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2pd4c" event={"ID":"64de3fa9-b45e-4425-8706-990230e0dcc9","Type":"ContainerStarted","Data":"0c2aaff65713cd1034a4f01c67463c450e84e89ce3fe48b29217e24f371f241b"}
Jul 28 16:04:39.683304 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:39.683271    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2pd4c" event={"ID":"64de3fa9-b45e-4425-8706-990230e0dcc9","Type":"ContainerStarted","Data":"3fb77fda445c5a9cc03c2325fcdf41acf2b04f290eb3cb5f10bcc74009b36726"}
Jul 28 16:04:39.683632 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:39.683345    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2pd4c" event={"ID":"64de3fa9-b45e-4425-8706-990230e0dcc9","Type":"ContainerStarted","Data":"e3282a45e11952296facf88331e38d1c9789e07391ea298519ab6640499fadf3"}
Jul 28 16:04:39.683632 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:39.683402    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:04:39.708828 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:39.708779    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2pd4c" podStartSLOduration=129.490996301 podStartE2EDuration="2m11.708767356s" totalImagesPullingTime="2.217771055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:39.707951129 +0000 UTC m=+166.266507689" watchObservedRunningTime="2026-07-28 16:04:39.708767356 +0000 UTC m=+166.267323898"
Jul 28 16:04:49.688618 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:04:49.688547    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2pd4c"
Jul 28 16:06:53.866323 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:06:53.866294    2581 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 16:07:07.641625 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:07.641589    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg"]
Jul 28 16:07:07.644869 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:07.644853    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg"
Jul 28 16:07:07.648531 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:07.648510    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 16:07:07.648640 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:07.648601    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 16:07:07.648705 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:07.648690    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-8sdm2\""
Jul 28 16:07:07.648844 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:07.648824    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 16:07:07.657428 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:07.657407    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg"]
Jul 28 16:07:07.710731 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:07.710701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/dd386c2e-7d9b-46e7-9b97-b25555d5cb70-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg\" (UID: \"dd386c2e-7d9b-46e7-9b97-b25555d5cb70\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg"
Jul 28 16:07:07.710874 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:07.710766    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftg6t\" (UniqueName: \"kubernetes.io/projected/dd386c2e-7d9b-46e7-9b97-b25555d5cb70-kube-api-access-ftg6t\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg\" (UID: \"dd386c2e-7d9b-46e7-9b97-b25555d5cb70\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg"
Jul 28 16:07:07.811751 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:07.811718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftg6t\" (UniqueName: \"kubernetes.io/projected/dd386c2e-7d9b-46e7-9b97-b25555d5cb70-kube-api-access-ftg6t\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg\" (UID: \"dd386c2e-7d9b-46e7-9b97-b25555d5cb70\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg"
Jul 28 16:07:07.811875 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:07.811760    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/dd386c2e-7d9b-46e7-9b97-b25555d5cb70-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg\" (UID: \"dd386c2e-7d9b-46e7-9b97-b25555d5cb70\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg"
Jul 28 16:07:07.814102 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:07.814082    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/dd386c2e-7d9b-46e7-9b97-b25555d5cb70-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg\" (UID: \"dd386c2e-7d9b-46e7-9b97-b25555d5cb70\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg"
Jul 28 16:07:07.826589 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:07.826562    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftg6t\" (UniqueName: \"kubernetes.io/projected/dd386c2e-7d9b-46e7-9b97-b25555d5cb70-kube-api-access-ftg6t\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg\" (UID: \"dd386c2e-7d9b-46e7-9b97-b25555d5cb70\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg"
Jul 28 16:07:07.954222 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:07.954188    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg"
Jul 28 16:07:08.084501 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:08.084464    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg"]
Jul 28 16:07:08.091045 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:07:08.090999    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd386c2e_7d9b_46e7_9b97_b25555d5cb70.slice/crio-398eef6851abdafedcc3f5dd1fdccde4185ee433c05a18687a2f02490da20beb WatchSource:0}: Error finding container 398eef6851abdafedcc3f5dd1fdccde4185ee433c05a18687a2f02490da20beb: Status 404 returned error can't find the container with id 398eef6851abdafedcc3f5dd1fdccde4185ee433c05a18687a2f02490da20beb
Jul 28 16:07:08.093366 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:08.093347    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:07:08.158295 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:08.158263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg" event={"ID":"dd386c2e-7d9b-46e7-9b97-b25555d5cb70","Type":"ContainerStarted","Data":"398eef6851abdafedcc3f5dd1fdccde4185ee433c05a18687a2f02490da20beb"}
Jul 28 16:07:17.190979 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:17.190939    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg" event={"ID":"dd386c2e-7d9b-46e7-9b97-b25555d5cb70","Type":"ContainerStarted","Data":"d29acc440ee4f38f71be1b0b3faf515eba8e4631dfaf36189aa62866cabf40cd"}
Jul 28 16:07:17.191357 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:17.191055    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg"
Jul 28 16:07:17.216287 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:17.216235    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg" podStartSLOduration=1.25804831 podStartE2EDuration="10.216219948s" totalImagesPullingTime="8.958171638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:07:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:07:17.215158087 +0000 UTC m=+323.773714642" watchObservedRunningTime="2026-07-28 16:07:17.216219948 +0000 UTC m=+323.774776491"
Jul 28 16:07:17.757984 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:17.757949    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-9t988"]
Jul 28 16:07:17.761027 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:17.760993    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-9t988"
Jul 28 16:07:17.763828 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:17.763796    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 28 16:07:17.764081 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:17.764065    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 16:07:17.764145 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:17.764127    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-gk8nr\""
Jul 28 16:07:17.775010 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:17.774989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2cad6342-a717-42ff-814e-9aa17176a1aa-certificates\") pod \"keda-operator-77f648d694-9t988\" (UID: \"2cad6342-a717-42ff-814e-9aa17176a1aa\") " pod="openshift-keda/keda-operator-77f648d694-9t988"
Jul 28 16:07:17.775119 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:17.775039    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxrb4\" (UniqueName: \"kubernetes.io/projected/2cad6342-a717-42ff-814e-9aa17176a1aa-kube-api-access-pxrb4\") pod \"keda-operator-77f648d694-9t988\" (UID: \"2cad6342-a717-42ff-814e-9aa17176a1aa\") " pod="openshift-keda/keda-operator-77f648d694-9t988"
Jul 28 16:07:17.775119 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:17.775094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2cad6342-a717-42ff-814e-9aa17176a1aa-cabundle0\") pod \"keda-operator-77f648d694-9t988\" (UID: \"2cad6342-a717-42ff-814e-9aa17176a1aa\") " pod="openshift-keda/keda-operator-77f648d694-9t988"
Jul 28 16:07:17.835509 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:17.833324    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-9t988"]
Jul 28 16:07:17.875799 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:17.875766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxrb4\" (UniqueName: \"kubernetes.io/projected/2cad6342-a717-42ff-814e-9aa17176a1aa-kube-api-access-pxrb4\") pod \"keda-operator-77f648d694-9t988\" (UID: \"2cad6342-a717-42ff-814e-9aa17176a1aa\") " pod="openshift-keda/keda-operator-77f648d694-9t988"
Jul 28 16:07:17.875934 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:17.875809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2cad6342-a717-42ff-814e-9aa17176a1aa-cabundle0\") pod \"keda-operator-77f648d694-9t988\" (UID: \"2cad6342-a717-42ff-814e-9aa17176a1aa\") " pod="openshift-keda/keda-operator-77f648d694-9t988"
Jul 28 16:07:17.875934 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:17.875850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2cad6342-a717-42ff-814e-9aa17176a1aa-certificates\") pod \"keda-operator-77f648d694-9t988\" (UID: \"2cad6342-a717-42ff-814e-9aa17176a1aa\") " pod="openshift-keda/keda-operator-77f648d694-9t988"
Jul 28 16:07:17.876034 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:07:17.875940    2581 secret.go:281] references non-existent secret key: ca.crt
Jul 28 16:07:17.876034 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:07:17.875952    2581 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 16:07:17.876034 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:07:17.875959    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-9t988: references non-existent secret key: ca.crt
Jul 28 16:07:17.876034 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:07:17.876008    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cad6342-a717-42ff-814e-9aa17176a1aa-certificates podName:2cad6342-a717-42ff-814e-9aa17176a1aa nodeName:}" failed. No retries permitted until 2026-07-28 16:07:18.375990003 +0000 UTC m=+324.934546523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/2cad6342-a717-42ff-814e-9aa17176a1aa-certificates") pod "keda-operator-77f648d694-9t988" (UID: "2cad6342-a717-42ff-814e-9aa17176a1aa") : references non-existent secret key: ca.crt
Jul 28 16:07:17.876472 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:17.876453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2cad6342-a717-42ff-814e-9aa17176a1aa-cabundle0\") pod \"keda-operator-77f648d694-9t988\" (UID: \"2cad6342-a717-42ff-814e-9aa17176a1aa\") " pod="openshift-keda/keda-operator-77f648d694-9t988"
Jul 28 16:07:17.890973 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:17.890940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxrb4\" (UniqueName: \"kubernetes.io/projected/2cad6342-a717-42ff-814e-9aa17176a1aa-kube-api-access-pxrb4\") pod \"keda-operator-77f648d694-9t988\" (UID: \"2cad6342-a717-42ff-814e-9aa17176a1aa\") " pod="openshift-keda/keda-operator-77f648d694-9t988"
Jul 28 16:07:18.379928 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:18.379893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2cad6342-a717-42ff-814e-9aa17176a1aa-certificates\") pod \"keda-operator-77f648d694-9t988\" (UID: \"2cad6342-a717-42ff-814e-9aa17176a1aa\") " pod="openshift-keda/keda-operator-77f648d694-9t988"
Jul 28 16:07:18.380420 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:07:18.380069    2581 secret.go:281] references non-existent secret key: ca.crt
Jul 28 16:07:18.380420 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:07:18.380094    2581 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 16:07:18.380420 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:07:18.380107    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-9t988: references non-existent secret key: ca.crt
Jul 28 16:07:18.380420 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:07:18.380168    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cad6342-a717-42ff-814e-9aa17176a1aa-certificates podName:2cad6342-a717-42ff-814e-9aa17176a1aa nodeName:}" failed. No retries permitted until 2026-07-28 16:07:19.380149578 +0000 UTC m=+325.938706115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/2cad6342-a717-42ff-814e-9aa17176a1aa-certificates") pod "keda-operator-77f648d694-9t988" (UID: "2cad6342-a717-42ff-814e-9aa17176a1aa") : references non-existent secret key: ca.crt
Jul 28 16:07:19.387854 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:19.387821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2cad6342-a717-42ff-814e-9aa17176a1aa-certificates\") pod \"keda-operator-77f648d694-9t988\" (UID: \"2cad6342-a717-42ff-814e-9aa17176a1aa\") " pod="openshift-keda/keda-operator-77f648d694-9t988"
Jul 28 16:07:19.390328 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:19.390304    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2cad6342-a717-42ff-814e-9aa17176a1aa-certificates\") pod \"keda-operator-77f648d694-9t988\" (UID: \"2cad6342-a717-42ff-814e-9aa17176a1aa\") " pod="openshift-keda/keda-operator-77f648d694-9t988"
Jul 28 16:07:19.570545 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:19.570508    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-9t988"
Jul 28 16:07:19.703625 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:19.703594    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-9t988"]
Jul 28 16:07:19.706657 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:07:19.706625    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cad6342_a717_42ff_814e_9aa17176a1aa.slice/crio-6f4e6f9dc6cd44487486f0701f94ab782a7cab137ac673077bcadded46a8cee2 WatchSource:0}: Error finding container 6f4e6f9dc6cd44487486f0701f94ab782a7cab137ac673077bcadded46a8cee2: Status 404 returned error can't find the container with id 6f4e6f9dc6cd44487486f0701f94ab782a7cab137ac673077bcadded46a8cee2
Jul 28 16:07:20.203125 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:20.203094    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-9t988" event={"ID":"2cad6342-a717-42ff-814e-9aa17176a1aa","Type":"ContainerStarted","Data":"6f4e6f9dc6cd44487486f0701f94ab782a7cab137ac673077bcadded46a8cee2"}
Jul 28 16:07:23.214832 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:23.214799    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-9t988" event={"ID":"2cad6342-a717-42ff-814e-9aa17176a1aa","Type":"ContainerStarted","Data":"83eddb2a2650e8aa1aa0c5622d03000fe281c629f2c657097ebe705eb8001627"}
Jul 28 16:07:23.215297 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:23.214938    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-9t988"
Jul 28 16:07:23.231292 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:23.231195    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-9t988" podStartSLOduration=2.961287178 podStartE2EDuration="6.231181683s" totalImagesPullingTime="3.269894505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:07:23.229973212 +0000 UTC m=+329.788529757" watchObservedRunningTime="2026-07-28 16:07:23.231181683 +0000 UTC m=+329.789738229"
Jul 28 16:07:38.196999 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:38.196962    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nt4tg"
Jul 28 16:07:44.220204 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:07:44.220173    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-9t988"
Jul 28 16:08:35.420168 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:35.420139    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-njxlk"]
Jul 28 16:08:35.423004 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:35.422987    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-njxlk"
Jul 28 16:08:35.425262 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:35.425240    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 16:08:35.425514 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:35.425497    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 16:08:35.425572 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:35.425547    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 16:08:35.425753 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:35.425731    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4b7lq\""
Jul 28 16:08:35.433911 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:35.433887    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-njxlk"]
Jul 28 16:08:35.519566 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:35.519529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3255b5f2-5bb3-4e3f-a698-f78e1ec82891-data\") pod \"seaweedfs-64568bcd49-njxlk\" (UID: \"3255b5f2-5bb3-4e3f-a698-f78e1ec82891\") " pod="kserve/seaweedfs-64568bcd49-njxlk"
Jul 28 16:08:35.519566 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:35.519565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq88s\" (UniqueName: \"kubernetes.io/projected/3255b5f2-5bb3-4e3f-a698-f78e1ec82891-kube-api-access-jq88s\") pod \"seaweedfs-64568bcd49-njxlk\" (UID: \"3255b5f2-5bb3-4e3f-a698-f78e1ec82891\") " pod="kserve/seaweedfs-64568bcd49-njxlk"
Jul 28 16:08:35.620148 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:35.620092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3255b5f2-5bb3-4e3f-a698-f78e1ec82891-data\") pod \"seaweedfs-64568bcd49-njxlk\" (UID: \"3255b5f2-5bb3-4e3f-a698-f78e1ec82891\") " pod="kserve/seaweedfs-64568bcd49-njxlk"
Jul 28 16:08:35.620148 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:35.620151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq88s\" (UniqueName: \"kubernetes.io/projected/3255b5f2-5bb3-4e3f-a698-f78e1ec82891-kube-api-access-jq88s\") pod \"seaweedfs-64568bcd49-njxlk\" (UID: \"3255b5f2-5bb3-4e3f-a698-f78e1ec82891\") " pod="kserve/seaweedfs-64568bcd49-njxlk"
Jul 28 16:08:35.620583 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:35.620563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3255b5f2-5bb3-4e3f-a698-f78e1ec82891-data\") pod \"seaweedfs-64568bcd49-njxlk\" (UID: \"3255b5f2-5bb3-4e3f-a698-f78e1ec82891\") " pod="kserve/seaweedfs-64568bcd49-njxlk"
Jul 28 16:08:35.629550 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:35.629521    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq88s\" (UniqueName: \"kubernetes.io/projected/3255b5f2-5bb3-4e3f-a698-f78e1ec82891-kube-api-access-jq88s\") pod \"seaweedfs-64568bcd49-njxlk\" (UID: \"3255b5f2-5bb3-4e3f-a698-f78e1ec82891\") " pod="kserve/seaweedfs-64568bcd49-njxlk"
Jul 28 16:08:35.731766 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:35.731673    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-njxlk"
Jul 28 16:08:35.854301 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:35.854264    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-njxlk"]
Jul 28 16:08:35.857235 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:08:35.857204    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3255b5f2_5bb3_4e3f_a698_f78e1ec82891.slice/crio-60282ea43fbe48de0d6c3f96d4257fd9aa9edebe8d729dfe846dc58b30b25b1a WatchSource:0}: Error finding container 60282ea43fbe48de0d6c3f96d4257fd9aa9edebe8d729dfe846dc58b30b25b1a: Status 404 returned error can't find the container with id 60282ea43fbe48de0d6c3f96d4257fd9aa9edebe8d729dfe846dc58b30b25b1a
Jul 28 16:08:36.453151 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:36.453114    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-njxlk" event={"ID":"3255b5f2-5bb3-4e3f-a698-f78e1ec82891","Type":"ContainerStarted","Data":"60282ea43fbe48de0d6c3f96d4257fd9aa9edebe8d729dfe846dc58b30b25b1a"}
Jul 28 16:08:39.467644 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:39.467609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-njxlk" event={"ID":"3255b5f2-5bb3-4e3f-a698-f78e1ec82891","Type":"ContainerStarted","Data":"50cf7682290d073f1e40c454341823f2aa83ae96e896a2f40462960f52d3d475"}
Jul 28 16:08:39.468090 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:39.467725    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-njxlk"
Jul 28 16:08:39.485721 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:39.485666    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-njxlk" podStartSLOduration=1.249642026 podStartE2EDuration="4.485649991s" totalImagesPullingTime="3.236007965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:08:39.484249046 +0000 UTC m=+406.042805588" watchObservedRunningTime="2026-07-28 16:08:39.485649991 +0000 UTC m=+406.044206532"
Jul 28 16:08:45.473347 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:08:45.473320    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-njxlk"
Jul 28 16:09:53.289988 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.289952    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-7mjbz"]
Jul 28 16:09:53.292897 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.292879    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-7mjbz"
Jul 28 16:09:53.295143 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.295121    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 16:09:53.295323 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.295302    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-pvlk5\""
Jul 28 16:09:53.301214 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.301192    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-7mjbz"]
Jul 28 16:09:53.303883 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.303866    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-px782"]
Jul 28 16:09:53.306756 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.306737    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-px782"
Jul 28 16:09:53.309059 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.309037    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 16:09:53.309138 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.309044    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-rpjbz\""
Jul 28 16:09:53.316174 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.316153    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-px782"]
Jul 28 16:09:53.379007 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.378973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgk4w\" (UniqueName: \"kubernetes.io/projected/42c4dbdb-2654-4200-a10a-15791b5c9efc-kube-api-access-cgk4w\") pod \"odh-model-controller-75d9cdbddf-px782\" (UID: \"42c4dbdb-2654-4200-a10a-15791b5c9efc\") " pod="kserve/odh-model-controller-75d9cdbddf-px782"
Jul 28 16:09:53.379007 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.379007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8388b5-aff7-414a-904c-4f3f55ec4c08-tls-certs\") pod \"model-serving-api-7b954ffc95-7mjbz\" (UID: \"3b8388b5-aff7-414a-904c-4f3f55ec4c08\") " pod="kserve/model-serving-api-7b954ffc95-7mjbz"
Jul 28 16:09:53.379271 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.379107    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7ft7\" (UniqueName: \"kubernetes.io/projected/3b8388b5-aff7-414a-904c-4f3f55ec4c08-kube-api-access-m7ft7\") pod \"model-serving-api-7b954ffc95-7mjbz\" (UID: \"3b8388b5-aff7-414a-904c-4f3f55ec4c08\") " pod="kserve/model-serving-api-7b954ffc95-7mjbz"
Jul 28 16:09:53.379271 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.379200    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42c4dbdb-2654-4200-a10a-15791b5c9efc-cert\") pod \"odh-model-controller-75d9cdbddf-px782\" (UID: \"42c4dbdb-2654-4200-a10a-15791b5c9efc\") " pod="kserve/odh-model-controller-75d9cdbddf-px782"
Jul 28 16:09:53.480480 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.480447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgk4w\" (UniqueName: \"kubernetes.io/projected/42c4dbdb-2654-4200-a10a-15791b5c9efc-kube-api-access-cgk4w\") pod \"odh-model-controller-75d9cdbddf-px782\" (UID: \"42c4dbdb-2654-4200-a10a-15791b5c9efc\") " pod="kserve/odh-model-controller-75d9cdbddf-px782"
Jul 28 16:09:53.480480 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.480481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8388b5-aff7-414a-904c-4f3f55ec4c08-tls-certs\") pod \"model-serving-api-7b954ffc95-7mjbz\" (UID: \"3b8388b5-aff7-414a-904c-4f3f55ec4c08\") " pod="kserve/model-serving-api-7b954ffc95-7mjbz"
Jul 28 16:09:53.480724 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.480509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7ft7\" (UniqueName: \"kubernetes.io/projected/3b8388b5-aff7-414a-904c-4f3f55ec4c08-kube-api-access-m7ft7\") pod \"model-serving-api-7b954ffc95-7mjbz\" (UID: \"3b8388b5-aff7-414a-904c-4f3f55ec4c08\") " pod="kserve/model-serving-api-7b954ffc95-7mjbz"
Jul 28 16:09:53.480724 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.480546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42c4dbdb-2654-4200-a10a-15791b5c9efc-cert\") pod \"odh-model-controller-75d9cdbddf-px782\" (UID: \"42c4dbdb-2654-4200-a10a-15791b5c9efc\") " pod="kserve/odh-model-controller-75d9cdbddf-px782"
Jul 28 16:09:53.480724 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:09:53.480648    2581 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 28 16:09:53.480724 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:09:53.480708    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42c4dbdb-2654-4200-a10a-15791b5c9efc-cert podName:42c4dbdb-2654-4200-a10a-15791b5c9efc nodeName:}" failed. No retries permitted until 2026-07-28 16:09:53.980689159 +0000 UTC m=+480.539245693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/42c4dbdb-2654-4200-a10a-15791b5c9efc-cert") pod "odh-model-controller-75d9cdbddf-px782" (UID: "42c4dbdb-2654-4200-a10a-15791b5c9efc") : secret "odh-model-controller-webhook-cert" not found
Jul 28 16:09:53.482868 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.482845    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8388b5-aff7-414a-904c-4f3f55ec4c08-tls-certs\") pod \"model-serving-api-7b954ffc95-7mjbz\" (UID: \"3b8388b5-aff7-414a-904c-4f3f55ec4c08\") " pod="kserve/model-serving-api-7b954ffc95-7mjbz"
Jul 28 16:09:53.489936 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.489914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgk4w\" (UniqueName: \"kubernetes.io/projected/42c4dbdb-2654-4200-a10a-15791b5c9efc-kube-api-access-cgk4w\") pod \"odh-model-controller-75d9cdbddf-px782\" (UID: \"42c4dbdb-2654-4200-a10a-15791b5c9efc\") " pod="kserve/odh-model-controller-75d9cdbddf-px782"
Jul 28 16:09:53.490059 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.490040    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7ft7\" (UniqueName: \"kubernetes.io/projected/3b8388b5-aff7-414a-904c-4f3f55ec4c08-kube-api-access-m7ft7\") pod \"model-serving-api-7b954ffc95-7mjbz\" (UID: \"3b8388b5-aff7-414a-904c-4f3f55ec4c08\") " pod="kserve/model-serving-api-7b954ffc95-7mjbz"
Jul 28 16:09:53.603646 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.603564    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-7mjbz"
Jul 28 16:09:53.734384 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.734351    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-7mjbz"]
Jul 28 16:09:53.737010 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:09:53.736981    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8388b5_aff7_414a_904c_4f3f55ec4c08.slice/crio-ba6169ff5ef996ae8a2edc8ec753214a9520bebd542210460ffed8191deda4de WatchSource:0}: Error finding container ba6169ff5ef996ae8a2edc8ec753214a9520bebd542210460ffed8191deda4de: Status 404 returned error can't find the container with id ba6169ff5ef996ae8a2edc8ec753214a9520bebd542210460ffed8191deda4de
Jul 28 16:09:53.984371 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.984340    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42c4dbdb-2654-4200-a10a-15791b5c9efc-cert\") pod \"odh-model-controller-75d9cdbddf-px782\" (UID: \"42c4dbdb-2654-4200-a10a-15791b5c9efc\") " pod="kserve/odh-model-controller-75d9cdbddf-px782"
Jul 28 16:09:53.986386 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:53.986360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42c4dbdb-2654-4200-a10a-15791b5c9efc-cert\") pod \"odh-model-controller-75d9cdbddf-px782\" (UID: \"42c4dbdb-2654-4200-a10a-15791b5c9efc\") " pod="kserve/odh-model-controller-75d9cdbddf-px782"
Jul 28 16:09:54.217281 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:54.217243    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-rpjbz\""
Jul 28 16:09:54.225522 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:54.225493    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-px782"
Jul 28 16:09:54.381584 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:54.381547    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-px782"]
Jul 28 16:09:54.386758 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:09:54.386722    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c4dbdb_2654_4200_a10a_15791b5c9efc.slice/crio-2b5eec06ae7c37ccc0495c382e653d27a6d70084e69bc09296e5c74b2aa8594a WatchSource:0}: Error finding container 2b5eec06ae7c37ccc0495c382e653d27a6d70084e69bc09296e5c74b2aa8594a: Status 404 returned error can't find the container with id 2b5eec06ae7c37ccc0495c382e653d27a6d70084e69bc09296e5c74b2aa8594a
Jul 28 16:09:54.718288 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:54.718212    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-px782" event={"ID":"42c4dbdb-2654-4200-a10a-15791b5c9efc","Type":"ContainerStarted","Data":"2b5eec06ae7c37ccc0495c382e653d27a6d70084e69bc09296e5c74b2aa8594a"}
Jul 28 16:09:54.719928 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:54.719883    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-7mjbz" event={"ID":"3b8388b5-aff7-414a-904c-4f3f55ec4c08","Type":"ContainerStarted","Data":"ba6169ff5ef996ae8a2edc8ec753214a9520bebd542210460ffed8191deda4de"}
Jul 28 16:09:57.735471 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:57.735431    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-px782" event={"ID":"42c4dbdb-2654-4200-a10a-15791b5c9efc","Type":"ContainerStarted","Data":"c97291d5f32bb435335d03c39bf44d1fb761e4d679f9df20bf3dea966ef1b4c3"}
Jul 28 16:09:57.735874 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:57.735607    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-px782"
Jul 28 16:09:57.736994 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:57.736968    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-7mjbz" event={"ID":"3b8388b5-aff7-414a-904c-4f3f55ec4c08","Type":"ContainerStarted","Data":"55717cce8a6ce45bbd71b952786ebdb1dcc1c88b6dc533f78f8fc35c4bd03083"}
Jul 28 16:09:57.737103 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:57.737040    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-7mjbz"
Jul 28 16:09:57.754124 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:57.754070    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-px782" podStartSLOduration=1.790792347 podStartE2EDuration="4.754055482s" totalImagesPullingTime="2.963263135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:09:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:09:57.75286656 +0000 UTC m=+484.311423129" watchObservedRunningTime="2026-07-28 16:09:57.754055482 +0000 UTC m=+484.312612024"
Jul 28 16:09:57.768544 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:09:57.768483    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-7mjbz" podStartSLOduration=1.201431864 podStartE2EDuration="4.76846955s" totalImagesPullingTime="3.567037686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:09:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:09:57.767737328 +0000 UTC m=+484.326293898" watchObservedRunningTime="2026-07-28 16:09:57.76846955 +0000 UTC m=+484.327026121"
Jul 28 16:10:08.742804 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:10:08.742772    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-px782"
Jul 28 16:10:08.744667 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:10:08.744645    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-7mjbz"
Jul 28 16:17:09.028591 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.028516    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"]
Jul 28 16:17:09.031698 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.031679    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"
Jul 28 16:17:09.034276 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.034251    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:17:09.034276 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.034262    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-8b634-predictor-serving-cert\""
Jul 28 16:17:09.034419 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.034303    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-8b634-kube-rbac-proxy-sar-config\""
Jul 28 16:17:09.034419 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.034334    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:17:09.035150 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.035136    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-m9t97\""
Jul 28 16:17:09.041715 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.041691    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"]
Jul 28 16:17:09.142805 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.142772    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57d7g\" (UniqueName: \"kubernetes.io/projected/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-kube-api-access-57d7g\") pod \"message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw\" (UID: \"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6\") " pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"
Jul 28 16:17:09.142934 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.142818    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw\" (UID: \"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6\") " pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"
Jul 28 16:17:09.142934 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.142881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-proxy-tls\") pod \"message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw\" (UID: \"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6\") " pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"
Jul 28 16:17:09.244152 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.244124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-proxy-tls\") pod \"message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw\" (UID: \"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6\") " pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"
Jul 28 16:17:09.244261 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.244173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57d7g\" (UniqueName: \"kubernetes.io/projected/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-kube-api-access-57d7g\") pod \"message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw\" (UID: \"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6\") " pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"
Jul 28 16:17:09.244261 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.244209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw\" (UID: \"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6\") " pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"
Jul 28 16:17:09.244354 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:17:09.244257    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-serving-cert: secret "message-dumper-raw-8b634-predictor-serving-cert" not found
Jul 28 16:17:09.244354 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:17:09.244321    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-proxy-tls podName:13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6 nodeName:}" failed. No retries permitted until 2026-07-28 16:17:09.744301525 +0000 UTC m=+916.302858045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-proxy-tls") pod "message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw" (UID: "13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6") : secret "message-dumper-raw-8b634-predictor-serving-cert" not found
Jul 28 16:17:09.244840 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.244823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw\" (UID: \"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6\") " pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"
Jul 28 16:17:09.252731 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.252713    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57d7g\" (UniqueName: \"kubernetes.io/projected/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-kube-api-access-57d7g\") pod \"message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw\" (UID: \"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6\") " pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"
Jul 28 16:17:09.748178 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.748145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-proxy-tls\") pod \"message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw\" (UID: \"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6\") " pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"
Jul 28 16:17:09.750274 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.750256    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-proxy-tls\") pod \"message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw\" (UID: \"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6\") " pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"
Jul 28 16:17:09.941602 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:09.941564    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"
Jul 28 16:17:10.073035 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:10.072989    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"]
Jul 28 16:17:10.079522 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:10.079496    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:17:10.197582 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:10.197544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw" event={"ID":"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6","Type":"ContainerStarted","Data":"79ff4c37759c4ede7a574e923b8805207d7cc4a79c9a0a389c3c900a28346a86"}
Jul 28 16:17:12.207361 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:12.207307    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw" event={"ID":"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6","Type":"ContainerStarted","Data":"5458facf554f2fe66a233b01bf6eeba871da31afc07a154c57f1bc388158784b"}
Jul 28 16:17:14.216885 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:14.216845    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw" event={"ID":"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6","Type":"ContainerStarted","Data":"389d857d7d1fb75d0ea19e4f9c8ec970363e4b88d4a72e78b777f7031556ff28"}
Jul 28 16:17:14.217323 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:14.216909    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"
Jul 28 16:17:14.217323 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:14.216972    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"
Jul 28 16:17:14.218759 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:14.218742    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"
Jul 28 16:17:14.235668 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:14.235626    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw" podStartSLOduration=1.759872155 podStartE2EDuration="5.235614754s" totalImagesPullingTime="3.475742599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:17:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:17:14.23444041 +0000 UTC m=+920.792996960" watchObservedRunningTime="2026-07-28 16:17:14.235614754 +0000 UTC m=+920.794171297"
Jul 28 16:17:21.229176 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:17:21.229151    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"
Jul 28 16:19:04.059333 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:04.059303    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw_13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6/kserve-container/0.log"
Jul 28 16:19:04.459467 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:04.459437    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"]
Jul 28 16:19:04.459739 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:04.459710    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw" podUID="13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6" containerName="kserve-container" containerID="cri-o://5458facf554f2fe66a233b01bf6eeba871da31afc07a154c57f1bc388158784b" gracePeriod=30
Jul 28 16:19:04.459890 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:04.459749    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw" podUID="13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6" containerName="kube-rbac-proxy" containerID="cri-o://389d857d7d1fb75d0ea19e4f9c8ec970363e4b88d4a72e78b777f7031556ff28" gracePeriod=30
Jul 28 16:19:04.599250 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:04.599222    2581 generic.go:356] "Generic (PLEG): container finished" podID="13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6" containerID="389d857d7d1fb75d0ea19e4f9c8ec970363e4b88d4a72e78b777f7031556ff28" exitCode=2
Jul 28 16:19:04.599250 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:04.599249    2581 generic.go:356] "Generic (PLEG): container finished" podID="13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6" containerID="5458facf554f2fe66a233b01bf6eeba871da31afc07a154c57f1bc388158784b" exitCode=2
Jul 28 16:19:04.599385 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:04.599294    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw" event={"ID":"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6","Type":"ContainerDied","Data":"389d857d7d1fb75d0ea19e4f9c8ec970363e4b88d4a72e78b777f7031556ff28"}
Jul 28 16:19:04.599385 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:04.599351    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw" event={"ID":"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6","Type":"ContainerDied","Data":"5458facf554f2fe66a233b01bf6eeba871da31afc07a154c57f1bc388158784b"}
Jul 28 16:19:04.705822 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:04.705801    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"
Jul 28 16:19:04.864475 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:04.864419    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57d7g\" (UniqueName: \"kubernetes.io/projected/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-kube-api-access-57d7g\") pod \"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6\" (UID: \"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6\") "
Jul 28 16:19:04.864475 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:04.864448    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-proxy-tls\") pod \"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6\" (UID: \"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6\") "
Jul 28 16:19:04.864608 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:04.864478    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-kube-rbac-proxy-sar-config\") pod \"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6\" (UID: \"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6\") "
Jul 28 16:19:04.864841 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:04.864812    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6" (UID: "13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:19:04.866599 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:04.866571    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-kube-api-access-57d7g" (OuterVolumeSpecName: "kube-api-access-57d7g") pod "13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6" (UID: "13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6"). InnerVolumeSpecName "kube-api-access-57d7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:19:04.866679 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:04.866621    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6" (UID: "13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:19:04.965261 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:04.965236    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57d7g\" (UniqueName: \"kubernetes.io/projected/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-kube-api-access-57d7g\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:19:04.965261 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:04.965258    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-proxy-tls\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:19:04.965376 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:04.965268    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:19:05.604862 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:05.604826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw" event={"ID":"13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6","Type":"ContainerDied","Data":"79ff4c37759c4ede7a574e923b8805207d7cc4a79c9a0a389c3c900a28346a86"}
Jul 28 16:19:05.604862 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:05.604856    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"
Jul 28 16:19:05.605293 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:05.604901    2581 scope.go:117] "RemoveContainer" containerID="389d857d7d1fb75d0ea19e4f9c8ec970363e4b88d4a72e78b777f7031556ff28"
Jul 28 16:19:05.614288 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:05.614270    2581 scope.go:117] "RemoveContainer" containerID="5458facf554f2fe66a233b01bf6eeba871da31afc07a154c57f1bc388158784b"
Jul 28 16:19:05.628990 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:05.628963    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"]
Jul 28 16:19:05.633998 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:05.633972    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-8b634-predictor-6d7dddc6bc-n5qhw"]
Jul 28 16:19:05.982055 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:19:05.982004    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6" path="/var/lib/kubelet/pods/13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6/volumes"
Jul 28 16:25:05.310053 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.309998    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"]
Jul 28 16:25:05.312407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.310261    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6" containerName="kserve-container"
Jul 28 16:25:05.312407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.310272    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6" containerName="kserve-container"
Jul 28 16:25:05.312407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.310284    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6" containerName="kube-rbac-proxy"
Jul 28 16:25:05.312407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.310290    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6" containerName="kube-rbac-proxy"
Jul 28 16:25:05.312407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.310347    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6" containerName="kserve-container"
Jul 28 16:25:05.312407 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.310355    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="13861e3f-d77f-4a7a-a2f3-5e5301d0a8b6" containerName="kube-rbac-proxy"
Jul 28 16:25:05.313294 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.313279    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:05.315721 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.315696    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:25:05.315843 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.315704    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 16:25:05.315933 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.315916    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:25:05.316365 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.316341    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 16:25:05.316474 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.316386    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:25:05.316474 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.316415    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:25:05.316474 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.316462    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-m9t97\""
Jul 28 16:25:05.328105 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.328084    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"]
Jul 28 16:25:05.380330 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.380296    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn6cz\" (UniqueName: \"kubernetes.io/projected/f971a4e2-11e5-44c3-97cc-e270ee607749-kube-api-access-sn6cz\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-2sshh\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:05.380330 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.380335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f971a4e2-11e5-44c3-97cc-e270ee607749-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-2sshh\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:05.380552 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.380403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f971a4e2-11e5-44c3-97cc-e270ee607749-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-2sshh\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:05.380552 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.380419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f971a4e2-11e5-44c3-97cc-e270ee607749-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-2sshh\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:05.380552 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.380438    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f971a4e2-11e5-44c3-97cc-e270ee607749-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-2sshh\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:05.481635 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.481601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f971a4e2-11e5-44c3-97cc-e270ee607749-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-2sshh\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:05.481635 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.481634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f971a4e2-11e5-44c3-97cc-e270ee607749-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-2sshh\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:05.481899 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.481656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f971a4e2-11e5-44c3-97cc-e270ee607749-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-2sshh\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:05.481899 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.481675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn6cz\" (UniqueName: \"kubernetes.io/projected/f971a4e2-11e5-44c3-97cc-e270ee607749-kube-api-access-sn6cz\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-2sshh\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:05.481899 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.481816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f971a4e2-11e5-44c3-97cc-e270ee607749-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-2sshh\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:05.482196 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.482173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f971a4e2-11e5-44c3-97cc-e270ee607749-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-2sshh\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:05.482452 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.482434    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f971a4e2-11e5-44c3-97cc-e270ee607749-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-2sshh\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:05.482489 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.482438    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f971a4e2-11e5-44c3-97cc-e270ee607749-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-2sshh\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:05.484296 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.484278    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f971a4e2-11e5-44c3-97cc-e270ee607749-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-2sshh\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:05.491079 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.491059    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn6cz\" (UniqueName: \"kubernetes.io/projected/f971a4e2-11e5-44c3-97cc-e270ee607749-kube-api-access-sn6cz\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-2sshh\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:05.622917 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.622820    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:05.754709 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.754675    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"]
Jul 28 16:25:05.758068 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:25:05.758040    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf971a4e2_11e5_44c3_97cc_e270ee607749.slice/crio-7ea2946e84d2c061546d444e8425eae2149f14ad8f9cbfaca7493f775e51f38b WatchSource:0}: Error finding container 7ea2946e84d2c061546d444e8425eae2149f14ad8f9cbfaca7493f775e51f38b: Status 404 returned error can't find the container with id 7ea2946e84d2c061546d444e8425eae2149f14ad8f9cbfaca7493f775e51f38b
Jul 28 16:25:05.760035 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.759991    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:25:05.827149 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:05.827115    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" event={"ID":"f971a4e2-11e5-44c3-97cc-e270ee607749","Type":"ContainerStarted","Data":"7ea2946e84d2c061546d444e8425eae2149f14ad8f9cbfaca7493f775e51f38b"}
Jul 28 16:25:09.844567 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:09.844486    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" event={"ID":"f971a4e2-11e5-44c3-97cc-e270ee607749","Type":"ContainerStarted","Data":"b67056fcf8a11500b9335c388083ea5c655723766bc0e40aa126878296ea56e0"}
Jul 28 16:25:13.860004 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:13.859971    2581 generic.go:356] "Generic (PLEG): container finished" podID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerID="b67056fcf8a11500b9335c388083ea5c655723766bc0e40aa126878296ea56e0" exitCode=0
Jul 28 16:25:13.860397 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:13.860044    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" event={"ID":"f971a4e2-11e5-44c3-97cc-e270ee607749","Type":"ContainerDied","Data":"b67056fcf8a11500b9335c388083ea5c655723766bc0e40aa126878296ea56e0"}
Jul 28 16:25:28.924239 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:28.924201    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" event={"ID":"f971a4e2-11e5-44c3-97cc-e270ee607749","Type":"ContainerStarted","Data":"275ad348fa3f4df06cbf4c13ca0de181eb54fea318407ca21f239cb4aba23d96"}
Jul 28 16:25:28.924635 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:28.924294    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" event={"ID":"f971a4e2-11e5-44c3-97cc-e270ee607749","Type":"ContainerStarted","Data":"71c864ec4e5ba4eea6eb89ad85804d6e190b4dd13c6565552511176851950913"}
Jul 28 16:25:28.924635 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:28.924488    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:28.945581 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:28.945541    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" podStartSLOduration=2.198260431 podStartE2EDuration="23.945528597s" totalImagesPullingTime="18.053233668s" totalInitContainerRuntime="3.694034498s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:25:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:25:28.944212757 +0000 UTC m=+1415.502769299" watchObservedRunningTime="2026-07-28 16:25:28.945528597 +0000 UTC m=+1415.504085141"
Jul 28 16:25:29.930501 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:29.930468    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:29.931601 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:29.931574    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 28 16:25:30.934662 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:30.934624    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 28 16:25:35.939729 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:35.939695    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:25:35.940368 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:35.940339    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 28 16:25:45.940428 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:45.940386    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 28 16:25:55.940881 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:25:55.940837    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 28 16:26:05.940695 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:05.940657    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 28 16:26:15.940990 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:15.940949    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 28 16:26:25.940636 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:25.940601    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 28 16:26:35.940447 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:35.940406    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 28 16:26:45.941783 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:45.941748    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:26:55.734697 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:55.734666    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"]
Jul 28 16:26:55.735086 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:55.734924    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kserve-container" containerID="cri-o://71c864ec4e5ba4eea6eb89ad85804d6e190b4dd13c6565552511176851950913" gracePeriod=30
Jul 28 16:26:55.735086 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:55.734981    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kube-rbac-proxy" containerID="cri-o://275ad348fa3f4df06cbf4c13ca0de181eb54fea318407ca21f239cb4aba23d96" gracePeriod=30
Jul 28 16:26:55.935656 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:55.935624    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 28 16:26:55.941105 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:55.941079    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 28 16:26:56.228473 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:56.228449    2581 generic.go:356] "Generic (PLEG): container finished" podID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerID="275ad348fa3f4df06cbf4c13ca0de181eb54fea318407ca21f239cb4aba23d96" exitCode=2
Jul 28 16:26:56.228574 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:56.228509    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" event={"ID":"f971a4e2-11e5-44c3-97cc-e270ee607749","Type":"ContainerDied","Data":"275ad348fa3f4df06cbf4c13ca0de181eb54fea318407ca21f239cb4aba23d96"}
Jul 28 16:26:59.678811 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:59.678786    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:26:59.848647 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:59.848581    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sn6cz\" (UniqueName: \"kubernetes.io/projected/f971a4e2-11e5-44c3-97cc-e270ee607749-kube-api-access-sn6cz\") pod \"f971a4e2-11e5-44c3-97cc-e270ee607749\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") "
Jul 28 16:26:59.848647 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:59.848621    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f971a4e2-11e5-44c3-97cc-e270ee607749-kserve-provision-location\") pod \"f971a4e2-11e5-44c3-97cc-e270ee607749\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") "
Jul 28 16:26:59.848851 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:59.848656    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f971a4e2-11e5-44c3-97cc-e270ee607749-kube-rbac-proxy-sar-config\") pod \"f971a4e2-11e5-44c3-97cc-e270ee607749\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") "
Jul 28 16:26:59.848851 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:59.848695    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f971a4e2-11e5-44c3-97cc-e270ee607749-cabundle-cert\") pod \"f971a4e2-11e5-44c3-97cc-e270ee607749\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") "
Jul 28 16:26:59.848851 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:59.848743    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f971a4e2-11e5-44c3-97cc-e270ee607749-proxy-tls\") pod \"f971a4e2-11e5-44c3-97cc-e270ee607749\" (UID: \"f971a4e2-11e5-44c3-97cc-e270ee607749\") "
Jul 28 16:26:59.849038 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:59.848991    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f971a4e2-11e5-44c3-97cc-e270ee607749-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f971a4e2-11e5-44c3-97cc-e270ee607749" (UID: "f971a4e2-11e5-44c3-97cc-e270ee607749"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:26:59.849038 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:59.849006    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f971a4e2-11e5-44c3-97cc-e270ee607749-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f971a4e2-11e5-44c3-97cc-e270ee607749" (UID: "f971a4e2-11e5-44c3-97cc-e270ee607749"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:26:59.849147 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:59.849047    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f971a4e2-11e5-44c3-97cc-e270ee607749-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f971a4e2-11e5-44c3-97cc-e270ee607749" (UID: "f971a4e2-11e5-44c3-97cc-e270ee607749"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:26:59.850598 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:59.850574    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f971a4e2-11e5-44c3-97cc-e270ee607749-kube-api-access-sn6cz" (OuterVolumeSpecName: "kube-api-access-sn6cz") pod "f971a4e2-11e5-44c3-97cc-e270ee607749" (UID: "f971a4e2-11e5-44c3-97cc-e270ee607749"). InnerVolumeSpecName "kube-api-access-sn6cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:26:59.850705 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:59.850666    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f971a4e2-11e5-44c3-97cc-e270ee607749-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f971a4e2-11e5-44c3-97cc-e270ee607749" (UID: "f971a4e2-11e5-44c3-97cc-e270ee607749"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:26:59.949582 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:59.949560    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f971a4e2-11e5-44c3-97cc-e270ee607749-cabundle-cert\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:26:59.949582 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:59.949579    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f971a4e2-11e5-44c3-97cc-e270ee607749-proxy-tls\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:26:59.949709 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:59.949588    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn6cz\" (UniqueName: \"kubernetes.io/projected/f971a4e2-11e5-44c3-97cc-e270ee607749-kube-api-access-sn6cz\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:26:59.949709 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:59.949597    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f971a4e2-11e5-44c3-97cc-e270ee607749-kserve-provision-location\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:26:59.949709 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:26:59.949605    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f971a4e2-11e5-44c3-97cc-e270ee607749-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:27:00.243390 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.243349    2581 generic.go:356] "Generic (PLEG): container finished" podID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerID="71c864ec4e5ba4eea6eb89ad85804d6e190b4dd13c6565552511176851950913" exitCode=0
Jul 28 16:27:00.243557 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.243405    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"
Jul 28 16:27:00.243557 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.243420    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" event={"ID":"f971a4e2-11e5-44c3-97cc-e270ee607749","Type":"ContainerDied","Data":"71c864ec4e5ba4eea6eb89ad85804d6e190b4dd13c6565552511176851950913"}
Jul 28 16:27:00.243557 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.243492    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh" event={"ID":"f971a4e2-11e5-44c3-97cc-e270ee607749","Type":"ContainerDied","Data":"7ea2946e84d2c061546d444e8425eae2149f14ad8f9cbfaca7493f775e51f38b"}
Jul 28 16:27:00.243557 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.243533    2581 scope.go:117] "RemoveContainer" containerID="275ad348fa3f4df06cbf4c13ca0de181eb54fea318407ca21f239cb4aba23d96"
Jul 28 16:27:00.256334 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.256313    2581 scope.go:117] "RemoveContainer" containerID="71c864ec4e5ba4eea6eb89ad85804d6e190b4dd13c6565552511176851950913"
Jul 28 16:27:00.263999 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.263977    2581 scope.go:117] "RemoveContainer" containerID="b67056fcf8a11500b9335c388083ea5c655723766bc0e40aa126878296ea56e0"
Jul 28 16:27:00.265033 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.265000    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"]
Jul 28 16:27:00.269420 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.269398    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-2sshh"]
Jul 28 16:27:00.271339 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.271324    2581 scope.go:117] "RemoveContainer" containerID="275ad348fa3f4df06cbf4c13ca0de181eb54fea318407ca21f239cb4aba23d96"
Jul 28 16:27:00.271616 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:27:00.271595    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"275ad348fa3f4df06cbf4c13ca0de181eb54fea318407ca21f239cb4aba23d96\": container with ID starting with 275ad348fa3f4df06cbf4c13ca0de181eb54fea318407ca21f239cb4aba23d96 not found: ID does not exist" containerID="275ad348fa3f4df06cbf4c13ca0de181eb54fea318407ca21f239cb4aba23d96"
Jul 28 16:27:00.271686 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.271628    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"275ad348fa3f4df06cbf4c13ca0de181eb54fea318407ca21f239cb4aba23d96"} err="failed to get container status \"275ad348fa3f4df06cbf4c13ca0de181eb54fea318407ca21f239cb4aba23d96\": rpc error: code = NotFound desc = could not find container \"275ad348fa3f4df06cbf4c13ca0de181eb54fea318407ca21f239cb4aba23d96\": container with ID starting with 275ad348fa3f4df06cbf4c13ca0de181eb54fea318407ca21f239cb4aba23d96 not found: ID does not exist"
Jul 28 16:27:00.271686 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.271653    2581 scope.go:117] "RemoveContainer" containerID="71c864ec4e5ba4eea6eb89ad85804d6e190b4dd13c6565552511176851950913"
Jul 28 16:27:00.271895 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:27:00.271879    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c864ec4e5ba4eea6eb89ad85804d6e190b4dd13c6565552511176851950913\": container with ID starting with 71c864ec4e5ba4eea6eb89ad85804d6e190b4dd13c6565552511176851950913 not found: ID does not exist" containerID="71c864ec4e5ba4eea6eb89ad85804d6e190b4dd13c6565552511176851950913"
Jul 28 16:27:00.271941 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.271899    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c864ec4e5ba4eea6eb89ad85804d6e190b4dd13c6565552511176851950913"} err="failed to get container status \"71c864ec4e5ba4eea6eb89ad85804d6e190b4dd13c6565552511176851950913\": rpc error: code = NotFound desc = could not find container \"71c864ec4e5ba4eea6eb89ad85804d6e190b4dd13c6565552511176851950913\": container with ID starting with 71c864ec4e5ba4eea6eb89ad85804d6e190b4dd13c6565552511176851950913 not found: ID does not exist"
Jul 28 16:27:00.271941 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.271914    2581 scope.go:117] "RemoveContainer" containerID="b67056fcf8a11500b9335c388083ea5c655723766bc0e40aa126878296ea56e0"
Jul 28 16:27:00.272141 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:27:00.272123    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b67056fcf8a11500b9335c388083ea5c655723766bc0e40aa126878296ea56e0\": container with ID starting with b67056fcf8a11500b9335c388083ea5c655723766bc0e40aa126878296ea56e0 not found: ID does not exist" containerID="b67056fcf8a11500b9335c388083ea5c655723766bc0e40aa126878296ea56e0"
Jul 28 16:27:00.272189 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.272144    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b67056fcf8a11500b9335c388083ea5c655723766bc0e40aa126878296ea56e0"} err="failed to get container status \"b67056fcf8a11500b9335c388083ea5c655723766bc0e40aa126878296ea56e0\": rpc error: code = NotFound desc = could not find container \"b67056fcf8a11500b9335c388083ea5c655723766bc0e40aa126878296ea56e0\": container with ID starting with b67056fcf8a11500b9335c388083ea5c655723766bc0e40aa126878296ea56e0 not found: ID does not exist"
Jul 28 16:27:00.826142 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.826108    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"]
Jul 28 16:27:00.826537 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.826368    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="storage-initializer"
Jul 28 16:27:00.826537 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.826380    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="storage-initializer"
Jul 28 16:27:00.826537 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.826394    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kserve-container"
Jul 28 16:27:00.826537 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.826400    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kserve-container"
Jul 28 16:27:00.826537 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.826413    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kube-rbac-proxy"
Jul 28 16:27:00.826537 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.826418    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kube-rbac-proxy"
Jul 28 16:27:00.826537 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.826460    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kserve-container"
Jul 28 16:27:00.826537 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.826467    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" containerName="kube-rbac-proxy"
Jul 28 16:27:00.830567 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.830548    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:00.833667 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.833650    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:27:00.833773 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.833739    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:27:00.833844 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.833784    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 16:27:00.833915 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.833900    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:27:00.834043 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.834006    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-m9t97\""
Jul 28 16:27:00.834139 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.834092    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:27:00.834235 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.834220    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 16:27:00.852480 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.852457    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"]
Jul 28 16:27:00.853461 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.853442    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"]
Jul 28 16:27:00.856522 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.856504    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:00.859695 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.859678    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 28 16:27:00.869696 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.869675    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"]
Jul 28 16:27:00.957257 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.957230    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-wv8d6\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:00.957363 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.957265    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc2017ae-1f90-4daa-b6f1-7c10224de29c-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:00.957363 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.957299    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc2017ae-1f90-4daa-b6f1-7c10224de29c-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:00.957363 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.957340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-wv8d6\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:00.957530 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.957376    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g724l\" (UniqueName: \"kubernetes.io/projected/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-kube-api-access-g724l\") pod \"isvc-canary-rollback-predictor-b4645bb5c-wv8d6\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:00.957530 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.957402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc2017ae-1f90-4daa-b6f1-7c10224de29c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:00.957530 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.957427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkm5x\" (UniqueName: \"kubernetes.io/projected/fc2017ae-1f90-4daa-b6f1-7c10224de29c-kube-api-access-xkm5x\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:00.957530 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.957473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc2017ae-1f90-4daa-b6f1-7c10224de29c-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:00.957530 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.957512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-wv8d6\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:00.957709 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:00.957553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-wv8d6\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:01.058675 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.058633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-wv8d6\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:01.058675 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.058680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g724l\" (UniqueName: \"kubernetes.io/projected/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-kube-api-access-g724l\") pod \"isvc-canary-rollback-predictor-b4645bb5c-wv8d6\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:01.058858 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.058707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc2017ae-1f90-4daa-b6f1-7c10224de29c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:01.058858 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.058757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkm5x\" (UniqueName: \"kubernetes.io/projected/fc2017ae-1f90-4daa-b6f1-7c10224de29c-kube-api-access-xkm5x\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:01.058858 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.058801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc2017ae-1f90-4daa-b6f1-7c10224de29c-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:01.058858 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.058847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-wv8d6\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:01.059168 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.058962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-wv8d6\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:01.059168 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.059070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-wv8d6\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:01.059168 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.059100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc2017ae-1f90-4daa-b6f1-7c10224de29c-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:01.059168 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.059135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc2017ae-1f90-4daa-b6f1-7c10224de29c-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:01.059354 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.059270    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-wv8d6\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:01.059470 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.059448    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc2017ae-1f90-4daa-b6f1-7c10224de29c-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:01.059595 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.059575    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc2017ae-1f90-4daa-b6f1-7c10224de29c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:01.059753 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.059733    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-wv8d6\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:01.059789 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.059732    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-wv8d6\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:01.059821 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.059785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc2017ae-1f90-4daa-b6f1-7c10224de29c-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:01.061189 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.061173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-wv8d6\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:01.061610 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.061595    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc2017ae-1f90-4daa-b6f1-7c10224de29c-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:01.070471 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.070444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkm5x\" (UniqueName: \"kubernetes.io/projected/fc2017ae-1f90-4daa-b6f1-7c10224de29c-kube-api-access-xkm5x\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:01.070657 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.070640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g724l\" (UniqueName: \"kubernetes.io/projected/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-kube-api-access-g724l\") pod \"isvc-canary-rollback-predictor-b4645bb5c-wv8d6\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:01.140782 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.140732    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:01.165085 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.165066    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:01.281244 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.281216    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"]
Jul 28 16:27:01.283930 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:27:01.283902    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8dbc806_31ac_41b9_82f6_eb0a3e54ddc3.slice/crio-c0d10a8644f615663e18a242e160093cf30a2b2168c09f67b7656bb17d439dc8 WatchSource:0}: Error finding container c0d10a8644f615663e18a242e160093cf30a2b2168c09f67b7656bb17d439dc8: Status 404 returned error can't find the container with id c0d10a8644f615663e18a242e160093cf30a2b2168c09f67b7656bb17d439dc8
Jul 28 16:27:01.303318 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.303295    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"]
Jul 28 16:27:01.310906 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:27:01.310885    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2017ae_1f90_4daa_b6f1_7c10224de29c.slice/crio-10cd01f09ca6d827c497dad6da3b4282c78566dbafb08e3990162c7ff4fdbd13 WatchSource:0}: Error finding container 10cd01f09ca6d827c497dad6da3b4282c78566dbafb08e3990162c7ff4fdbd13: Status 404 returned error can't find the container with id 10cd01f09ca6d827c497dad6da3b4282c78566dbafb08e3990162c7ff4fdbd13
Jul 28 16:27:01.981626 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:01.981583    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f971a4e2-11e5-44c3-97cc-e270ee607749" path="/var/lib/kubelet/pods/f971a4e2-11e5-44c3-97cc-e270ee607749/volumes"
Jul 28 16:27:02.254357 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:02.254267    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" event={"ID":"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3","Type":"ContainerStarted","Data":"ee227ec7b1884383f2e0febef0c2d1bd67c9aace7538cdb03a1145a67e2c16e3"}
Jul 28 16:27:02.254357 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:02.254344    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" event={"ID":"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3","Type":"ContainerStarted","Data":"c0d10a8644f615663e18a242e160093cf30a2b2168c09f67b7656bb17d439dc8"}
Jul 28 16:27:02.255664 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:02.255641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" event={"ID":"fc2017ae-1f90-4daa-b6f1-7c10224de29c","Type":"ContainerStarted","Data":"ba0e3b9f472682eebdb7fe2b8db7d8d935e0dea6e890d71be560bfdc6f7c56a3"}
Jul 28 16:27:02.255804 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:02.255696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" event={"ID":"fc2017ae-1f90-4daa-b6f1-7c10224de29c","Type":"ContainerStarted","Data":"10cd01f09ca6d827c497dad6da3b4282c78566dbafb08e3990162c7ff4fdbd13"}
Jul 28 16:27:05.267684 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:05.267647    2581 generic.go:356] "Generic (PLEG): container finished" podID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerID="ba0e3b9f472682eebdb7fe2b8db7d8d935e0dea6e890d71be560bfdc6f7c56a3" exitCode=0
Jul 28 16:27:05.268091 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:05.267723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" event={"ID":"fc2017ae-1f90-4daa-b6f1-7c10224de29c","Type":"ContainerDied","Data":"ba0e3b9f472682eebdb7fe2b8db7d8d935e0dea6e890d71be560bfdc6f7c56a3"}
Jul 28 16:27:05.269198 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:05.269179    2581 generic.go:356] "Generic (PLEG): container finished" podID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerID="ee227ec7b1884383f2e0febef0c2d1bd67c9aace7538cdb03a1145a67e2c16e3" exitCode=0
Jul 28 16:27:05.269295 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:05.269227    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" event={"ID":"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3","Type":"ContainerDied","Data":"ee227ec7b1884383f2e0febef0c2d1bd67c9aace7538cdb03a1145a67e2c16e3"}
Jul 28 16:27:06.275576 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:06.275543    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" event={"ID":"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3","Type":"ContainerStarted","Data":"7d8c2c870c868a29288f5d36d9671a843d781d5fbb14127f59d3fa2fc6ade6e9"}
Jul 28 16:27:06.276048 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:06.275603    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" event={"ID":"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3","Type":"ContainerStarted","Data":"b3826d8b6c246c68982a865e54985648f83ec54f09b17cfae47a7fdf2e651b82"}
Jul 28 16:27:06.276048 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:06.275808    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:06.277549 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:06.277525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" event={"ID":"fc2017ae-1f90-4daa-b6f1-7c10224de29c","Type":"ContainerStarted","Data":"029b5241f907cb1f7d6af9a41e694179dc6283f47ce1f69d6912116d536e00dc"}
Jul 28 16:27:06.277665 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:06.277582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" event={"ID":"fc2017ae-1f90-4daa-b6f1-7c10224de29c","Type":"ContainerStarted","Data":"d475336661ba2d6fd50a67f6c934be897a5ee16f5e6768a23b043f27bc02d58b"}
Jul 28 16:27:06.277845 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:06.277827    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:06.277914 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:06.277877    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:06.279192 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:06.279166    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 28 16:27:06.303309 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:06.303267    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" podStartSLOduration=2.7348425819999997 podStartE2EDuration="6.303254152s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.56841157s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:27:06.302584801 +0000 UTC m=+1512.861141343" watchObservedRunningTime="2026-07-28 16:27:06.303254152 +0000 UTC m=+1512.861810694"
Jul 28 16:27:06.329487 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:06.329446    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" podStartSLOduration=2.523086941 podStartE2EDuration="6.32943394s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.806346999s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:27:06.326742684 +0000 UTC m=+1512.885299225" watchObservedRunningTime="2026-07-28 16:27:06.32943394 +0000 UTC m=+1512.887990481"
Jul 28 16:27:07.288753 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:07.288256    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:07.289297 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:07.288759    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 16:27:07.289297 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:07.289010    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 28 16:27:08.289388 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:08.289349    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 16:27:12.290454 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:12.290422    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:27:12.290978 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:12.290950    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 28 16:27:13.293676 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:13.293647    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:27:13.294327 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:13.294299    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 16:27:22.290987 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:22.290891    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 28 16:27:23.294614 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:23.294574    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 16:27:32.290969 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:32.290930    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 28 16:27:33.295149 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:33.295109    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 16:27:42.291597 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:42.291562    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 28 16:27:43.294714 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:43.294672    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 16:27:52.291759 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:52.291725    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 28 16:27:53.294244 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:27:53.294202    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 16:28:02.291353 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:02.291312    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 28 16:28:03.294405 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:03.294364    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 16:28:12.291594 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:12.291560    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 28 16:28:13.295191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:13.295150    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 16:28:21.982519 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:21.982479    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:28:23.294725 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:23.294698    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:28:30.780870 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:30.780836    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"]
Jul 28 16:28:30.781339 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:30.781171    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kserve-container" containerID="cri-o://d475336661ba2d6fd50a67f6c934be897a5ee16f5e6768a23b043f27bc02d58b" gracePeriod=30
Jul 28 16:28:30.781339 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:30.781243    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kube-rbac-proxy" containerID="cri-o://029b5241f907cb1f7d6af9a41e694179dc6283f47ce1f69d6912116d536e00dc" gracePeriod=30
Jul 28 16:28:31.597971 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:31.597932    2581 generic.go:356] "Generic (PLEG): container finished" podID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerID="029b5241f907cb1f7d6af9a41e694179dc6283f47ce1f69d6912116d536e00dc" exitCode=2
Jul 28 16:28:31.598160 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:31.598004    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" event={"ID":"fc2017ae-1f90-4daa-b6f1-7c10224de29c","Type":"ContainerDied","Data":"029b5241f907cb1f7d6af9a41e694179dc6283f47ce1f69d6912116d536e00dc"}
Jul 28 16:28:31.977666 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:31.977616    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 28 16:28:32.286314 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:32.286235    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Jul 28 16:28:34.613191 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:34.613161    2581 generic.go:356] "Generic (PLEG): container finished" podID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerID="d475336661ba2d6fd50a67f6c934be897a5ee16f5e6768a23b043f27bc02d58b" exitCode=0
Jul 28 16:28:34.613489 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:34.613233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" event={"ID":"fc2017ae-1f90-4daa-b6f1-7c10224de29c","Type":"ContainerDied","Data":"d475336661ba2d6fd50a67f6c934be897a5ee16f5e6768a23b043f27bc02d58b"}
Jul 28 16:28:34.730283 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:34.730260    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:28:34.804944 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:34.804878    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc2017ae-1f90-4daa-b6f1-7c10224de29c-kserve-provision-location\") pod \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") "
Jul 28 16:28:34.804944 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:34.804932    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc2017ae-1f90-4daa-b6f1-7c10224de29c-proxy-tls\") pod \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") "
Jul 28 16:28:34.805146 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:34.804960    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc2017ae-1f90-4daa-b6f1-7c10224de29c-kube-rbac-proxy-sar-config\") pod \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") "
Jul 28 16:28:34.805146 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:34.804987    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkm5x\" (UniqueName: \"kubernetes.io/projected/fc2017ae-1f90-4daa-b6f1-7c10224de29c-kube-api-access-xkm5x\") pod \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") "
Jul 28 16:28:34.805146 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:34.805040    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc2017ae-1f90-4daa-b6f1-7c10224de29c-cabundle-cert\") pod \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\" (UID: \"fc2017ae-1f90-4daa-b6f1-7c10224de29c\") "
Jul 28 16:28:34.805292 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:34.805157    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2017ae-1f90-4daa-b6f1-7c10224de29c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc2017ae-1f90-4daa-b6f1-7c10224de29c" (UID: "fc2017ae-1f90-4daa-b6f1-7c10224de29c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:28:34.805292 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:34.805257    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc2017ae-1f90-4daa-b6f1-7c10224de29c-kserve-provision-location\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:28:34.805386 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:34.805347    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc2017ae-1f90-4daa-b6f1-7c10224de29c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc2017ae-1f90-4daa-b6f1-7c10224de29c" (UID: "fc2017ae-1f90-4daa-b6f1-7c10224de29c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:28:34.805444 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:34.805420    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc2017ae-1f90-4daa-b6f1-7c10224de29c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc2017ae-1f90-4daa-b6f1-7c10224de29c" (UID: "fc2017ae-1f90-4daa-b6f1-7c10224de29c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:28:34.806962 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:34.806938    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2017ae-1f90-4daa-b6f1-7c10224de29c-kube-api-access-xkm5x" (OuterVolumeSpecName: "kube-api-access-xkm5x") pod "fc2017ae-1f90-4daa-b6f1-7c10224de29c" (UID: "fc2017ae-1f90-4daa-b6f1-7c10224de29c"). InnerVolumeSpecName "kube-api-access-xkm5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:28:34.807050 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:34.806991    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc2017ae-1f90-4daa-b6f1-7c10224de29c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc2017ae-1f90-4daa-b6f1-7c10224de29c" (UID: "fc2017ae-1f90-4daa-b6f1-7c10224de29c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:28:34.905696 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:34.905670    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc2017ae-1f90-4daa-b6f1-7c10224de29c-proxy-tls\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:28:34.905696 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:34.905692    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc2017ae-1f90-4daa-b6f1-7c10224de29c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:28:34.905822 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:34.905703    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkm5x\" (UniqueName: \"kubernetes.io/projected/fc2017ae-1f90-4daa-b6f1-7c10224de29c-kube-api-access-xkm5x\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:28:34.905822 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:34.905712    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc2017ae-1f90-4daa-b6f1-7c10224de29c-cabundle-cert\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:28:35.618988 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:35.618951    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66" event={"ID":"fc2017ae-1f90-4daa-b6f1-7c10224de29c","Type":"ContainerDied","Data":"10cd01f09ca6d827c497dad6da3b4282c78566dbafb08e3990162c7ff4fdbd13"}
Jul 28 16:28:35.618988 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:35.618970    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"
Jul 28 16:28:35.619550 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:35.619036    2581 scope.go:117] "RemoveContainer" containerID="029b5241f907cb1f7d6af9a41e694179dc6283f47ce1f69d6912116d536e00dc"
Jul 28 16:28:35.629980 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:35.628551    2581 scope.go:117] "RemoveContainer" containerID="d475336661ba2d6fd50a67f6c934be897a5ee16f5e6768a23b043f27bc02d58b"
Jul 28 16:28:35.636380 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:35.636364    2581 scope.go:117] "RemoveContainer" containerID="ba0e3b9f472682eebdb7fe2b8db7d8d935e0dea6e890d71be560bfdc6f7c56a3"
Jul 28 16:28:35.644975 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:35.644952    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"]
Jul 28 16:28:35.650583 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:35.650561    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-h5m66"]
Jul 28 16:28:35.981621 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:28:35.981581    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" path="/var/lib/kubelet/pods/fc2017ae-1f90-4daa-b6f1-7c10224de29c/volumes"
Jul 28 16:30:01.027075 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:01.027045    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"]
Jul 28 16:30:01.027594 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:01.027278    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kserve-container" containerID="cri-o://b3826d8b6c246c68982a865e54985648f83ec54f09b17cfae47a7fdf2e651b82" gracePeriod=30
Jul 28 16:30:01.027594 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:01.027380    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kube-rbac-proxy" containerID="cri-o://7d8c2c870c868a29288f5d36d9671a843d781d5fbb14127f59d3fa2fc6ade6e9" gracePeriod=30
Jul 28 16:30:01.926123 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:01.926090    2581 generic.go:356] "Generic (PLEG): container finished" podID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerID="7d8c2c870c868a29288f5d36d9671a843d781d5fbb14127f59d3fa2fc6ade6e9" exitCode=2
Jul 28 16:30:01.926292 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:01.926147    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" event={"ID":"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3","Type":"ContainerDied","Data":"7d8c2c870c868a29288f5d36d9671a843d781d5fbb14127f59d3fa2fc6ade6e9"}
Jul 28 16:30:03.290617 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:03.290570    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 28 16:30:03.294976 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:03.294950    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 16:30:05.571753 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.571727    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:30:05.622296 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.622223    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-proxy-tls\") pod \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") "
Jul 28 16:30:05.622296 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.622259    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-kube-rbac-proxy-sar-config\") pod \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") "
Jul 28 16:30:05.622296 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.622286    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-cabundle-cert\") pod \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") "
Jul 28 16:30:05.622507 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.622327    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g724l\" (UniqueName: \"kubernetes.io/projected/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-kube-api-access-g724l\") pod \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") "
Jul 28 16:30:05.622507 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.622376    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-kserve-provision-location\") pod \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\" (UID: \"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3\") "
Jul 28 16:30:05.622733 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.622703    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" (UID: "b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:30:05.622733 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.622724    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" (UID: "b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:30:05.622843 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.622707    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" (UID: "b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:30:05.624348 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.624326    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-kube-api-access-g724l" (OuterVolumeSpecName: "kube-api-access-g724l") pod "b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" (UID: "b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3"). InnerVolumeSpecName "kube-api-access-g724l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:30:05.624434 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.624352    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" (UID: "b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:30:05.723323 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.723286    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g724l\" (UniqueName: \"kubernetes.io/projected/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-kube-api-access-g724l\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:30:05.723323 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.723317    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-kserve-provision-location\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:30:05.723323 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.723328    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-proxy-tls\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:30:05.723538 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.723336    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:30:05.723538 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.723345    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3-cabundle-cert\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:30:05.942232 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.942197    2581 generic.go:356] "Generic (PLEG): container finished" podID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerID="b3826d8b6c246c68982a865e54985648f83ec54f09b17cfae47a7fdf2e651b82" exitCode=0
Jul 28 16:30:05.942393 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.942249    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"
Jul 28 16:30:05.942393 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.942255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" event={"ID":"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3","Type":"ContainerDied","Data":"b3826d8b6c246c68982a865e54985648f83ec54f09b17cfae47a7fdf2e651b82"}
Jul 28 16:30:05.942393 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.942333    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6" event={"ID":"b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3","Type":"ContainerDied","Data":"c0d10a8644f615663e18a242e160093cf30a2b2168c09f67b7656bb17d439dc8"}
Jul 28 16:30:05.942393 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.942361    2581 scope.go:117] "RemoveContainer" containerID="7d8c2c870c868a29288f5d36d9671a843d781d5fbb14127f59d3fa2fc6ade6e9"
Jul 28 16:30:05.950566 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.950549    2581 scope.go:117] "RemoveContainer" containerID="b3826d8b6c246c68982a865e54985648f83ec54f09b17cfae47a7fdf2e651b82"
Jul 28 16:30:05.957472 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.957451    2581 scope.go:117] "RemoveContainer" containerID="ee227ec7b1884383f2e0febef0c2d1bd67c9aace7538cdb03a1145a67e2c16e3"
Jul 28 16:30:05.964177 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.964163    2581 scope.go:117] "RemoveContainer" containerID="7d8c2c870c868a29288f5d36d9671a843d781d5fbb14127f59d3fa2fc6ade6e9"
Jul 28 16:30:05.964411 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:30:05.964394    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d8c2c870c868a29288f5d36d9671a843d781d5fbb14127f59d3fa2fc6ade6e9\": container with ID starting with 7d8c2c870c868a29288f5d36d9671a843d781d5fbb14127f59d3fa2fc6ade6e9 not found: ID does not exist" containerID="7d8c2c870c868a29288f5d36d9671a843d781d5fbb14127f59d3fa2fc6ade6e9"
Jul 28 16:30:05.964448 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.964419    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8c2c870c868a29288f5d36d9671a843d781d5fbb14127f59d3fa2fc6ade6e9"} err="failed to get container status \"7d8c2c870c868a29288f5d36d9671a843d781d5fbb14127f59d3fa2fc6ade6e9\": rpc error: code = NotFound desc = could not find container \"7d8c2c870c868a29288f5d36d9671a843d781d5fbb14127f59d3fa2fc6ade6e9\": container with ID starting with 7d8c2c870c868a29288f5d36d9671a843d781d5fbb14127f59d3fa2fc6ade6e9 not found: ID does not exist"
Jul 28 16:30:05.964448 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.964437    2581 scope.go:117] "RemoveContainer" containerID="b3826d8b6c246c68982a865e54985648f83ec54f09b17cfae47a7fdf2e651b82"
Jul 28 16:30:05.964720 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:30:05.964702    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3826d8b6c246c68982a865e54985648f83ec54f09b17cfae47a7fdf2e651b82\": container with ID starting with b3826d8b6c246c68982a865e54985648f83ec54f09b17cfae47a7fdf2e651b82 not found: ID does not exist" containerID="b3826d8b6c246c68982a865e54985648f83ec54f09b17cfae47a7fdf2e651b82"
Jul 28 16:30:05.964776 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.964726    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3826d8b6c246c68982a865e54985648f83ec54f09b17cfae47a7fdf2e651b82"} err="failed to get container status \"b3826d8b6c246c68982a865e54985648f83ec54f09b17cfae47a7fdf2e651b82\": rpc error: code = NotFound desc = could not find container \"b3826d8b6c246c68982a865e54985648f83ec54f09b17cfae47a7fdf2e651b82\": container with ID starting with b3826d8b6c246c68982a865e54985648f83ec54f09b17cfae47a7fdf2e651b82 not found: ID does not exist"
Jul 28 16:30:05.964776 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.964757    2581 scope.go:117] "RemoveContainer" containerID="ee227ec7b1884383f2e0febef0c2d1bd67c9aace7538cdb03a1145a67e2c16e3"
Jul 28 16:30:05.964957 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:30:05.964938    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee227ec7b1884383f2e0febef0c2d1bd67c9aace7538cdb03a1145a67e2c16e3\": container with ID starting with ee227ec7b1884383f2e0febef0c2d1bd67c9aace7538cdb03a1145a67e2c16e3 not found: ID does not exist" containerID="ee227ec7b1884383f2e0febef0c2d1bd67c9aace7538cdb03a1145a67e2c16e3"
Jul 28 16:30:05.965002 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.964962    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee227ec7b1884383f2e0febef0c2d1bd67c9aace7538cdb03a1145a67e2c16e3"} err="failed to get container status \"ee227ec7b1884383f2e0febef0c2d1bd67c9aace7538cdb03a1145a67e2c16e3\": rpc error: code = NotFound desc = could not find container \"ee227ec7b1884383f2e0febef0c2d1bd67c9aace7538cdb03a1145a67e2c16e3\": container with ID starting with ee227ec7b1884383f2e0febef0c2d1bd67c9aace7538cdb03a1145a67e2c16e3 not found: ID does not exist"
Jul 28 16:30:05.968776 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.968753    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"]
Jul 28 16:30:05.981771 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:05.981749    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-wv8d6"]
Jul 28 16:30:06.282680 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.282601    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"]
Jul 28 16:30:06.282856 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.282844    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="storage-initializer"
Jul 28 16:30:06.282897 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.282857    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="storage-initializer"
Jul 28 16:30:06.282897 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.282868    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kube-rbac-proxy"
Jul 28 16:30:06.282897 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.282874    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kube-rbac-proxy"
Jul 28 16:30:06.282897 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.282883    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kserve-container"
Jul 28 16:30:06.282897 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.282889    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kserve-container"
Jul 28 16:30:06.282897 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.282895    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kserve-container"
Jul 28 16:30:06.283091 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.282901    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kserve-container"
Jul 28 16:30:06.283091 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.282908    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kube-rbac-proxy"
Jul 28 16:30:06.283091 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.282913    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kube-rbac-proxy"
Jul 28 16:30:06.283091 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.282923    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="storage-initializer"
Jul 28 16:30:06.283091 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.282928    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="storage-initializer"
Jul 28 16:30:06.283091 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.282971    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kserve-container"
Jul 28 16:30:06.283091 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.282977    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" containerName="kube-rbac-proxy"
Jul 28 16:30:06.283091 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.282983    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kserve-container"
Jul 28 16:30:06.283091 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.282988    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc2017ae-1f90-4daa-b6f1-7c10224de29c" containerName="kube-rbac-proxy"
Jul 28 16:30:06.287488 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.287464    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:06.291335 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.291298    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:30:06.291335 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.291321    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:30:06.291496 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.291330    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-m9t97\""
Jul 28 16:30:06.292033 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.291997    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 16:30:06.292115 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.292037    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:30:06.292115 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.292067    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 28 16:30:06.292280 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.292267    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:30:06.299438 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.299414    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"]
Jul 28 16:30:06.301632 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.301612    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"]
Jul 28 16:30:06.304749 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.304724    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:06.306879 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.306861    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 28 16:30:06.321182 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.321156    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"]
Jul 28 16:30:06.327460 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.327440    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5pzl\" (UniqueName: \"kubernetes.io/projected/ff08653c-5659-4a8b-a721-f5484033bdc1-kube-api-access-f5pzl\") pod \"isvc-canary-stop-predictor-5bf5667989-pl6gq\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:06.327547 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.327472    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff08653c-5659-4a8b-a721-f5484033bdc1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5bf5667989-pl6gq\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:06.327594 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.327552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff08653c-5659-4a8b-a721-f5484033bdc1-proxy-tls\") pod \"isvc-canary-stop-predictor-5bf5667989-pl6gq\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:06.327594 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.327588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff08653c-5659-4a8b-a721-f5484033bdc1-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5bf5667989-pl6gq\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:06.327676 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.327606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff08653c-5659-4a8b-a721-f5484033bdc1-cabundle-cert\") pod \"isvc-canary-stop-predictor-5bf5667989-pl6gq\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:06.428591 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.428555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02a79e67-dc23-45ac-a629-23e3e317b5ca-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-tpk56\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:06.428591 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.428595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff08653c-5659-4a8b-a721-f5484033bdc1-proxy-tls\") pod \"isvc-canary-stop-predictor-5bf5667989-pl6gq\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:06.428824 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.428631    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff08653c-5659-4a8b-a721-f5484033bdc1-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5bf5667989-pl6gq\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:06.428824 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.428655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff08653c-5659-4a8b-a721-f5484033bdc1-cabundle-cert\") pod \"isvc-canary-stop-predictor-5bf5667989-pl6gq\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:06.428824 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.428684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xftgn\" (UniqueName: \"kubernetes.io/projected/02a79e67-dc23-45ac-a629-23e3e317b5ca-kube-api-access-xftgn\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-tpk56\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:06.428824 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.428723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5pzl\" (UniqueName: \"kubernetes.io/projected/ff08653c-5659-4a8b-a721-f5484033bdc1-kube-api-access-f5pzl\") pod \"isvc-canary-stop-predictor-5bf5667989-pl6gq\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:06.428824 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.428762    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff08653c-5659-4a8b-a721-f5484033bdc1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5bf5667989-pl6gq\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:06.428824 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.428790    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02a79e67-dc23-45ac-a629-23e3e317b5ca-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-tpk56\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:06.428824 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.428820    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a79e67-dc23-45ac-a629-23e3e317b5ca-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-tpk56\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:06.429215 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.428852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a79e67-dc23-45ac-a629-23e3e317b5ca-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-tpk56\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:06.429215 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.429073    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff08653c-5659-4a8b-a721-f5484033bdc1-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5bf5667989-pl6gq\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:06.429483 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.429461    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff08653c-5659-4a8b-a721-f5484033bdc1-cabundle-cert\") pod \"isvc-canary-stop-predictor-5bf5667989-pl6gq\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:06.429553 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.429499    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff08653c-5659-4a8b-a721-f5484033bdc1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5bf5667989-pl6gq\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:06.431079 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.431055    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff08653c-5659-4a8b-a721-f5484033bdc1-proxy-tls\") pod \"isvc-canary-stop-predictor-5bf5667989-pl6gq\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:06.441693 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.441664    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5pzl\" (UniqueName: \"kubernetes.io/projected/ff08653c-5659-4a8b-a721-f5484033bdc1-kube-api-access-f5pzl\") pod \"isvc-canary-stop-predictor-5bf5667989-pl6gq\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:06.529525 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.529494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02a79e67-dc23-45ac-a629-23e3e317b5ca-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-tpk56\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:06.529666 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.529537    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xftgn\" (UniqueName: \"kubernetes.io/projected/02a79e67-dc23-45ac-a629-23e3e317b5ca-kube-api-access-xftgn\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-tpk56\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:06.529666 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.529569    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02a79e67-dc23-45ac-a629-23e3e317b5ca-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-tpk56\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:06.529666 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.529596    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a79e67-dc23-45ac-a629-23e3e317b5ca-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-tpk56\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:06.529666 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.529618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a79e67-dc23-45ac-a629-23e3e317b5ca-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-tpk56\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:06.529928 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.529903    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02a79e67-dc23-45ac-a629-23e3e317b5ca-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-tpk56\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:06.530312 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.530285    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a79e67-dc23-45ac-a629-23e3e317b5ca-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-tpk56\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:06.530401 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.530311    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02a79e67-dc23-45ac-a629-23e3e317b5ca-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-tpk56\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:06.531852 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.531826    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a79e67-dc23-45ac-a629-23e3e317b5ca-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-tpk56\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:06.539314 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.539262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xftgn\" (UniqueName: \"kubernetes.io/projected/02a79e67-dc23-45ac-a629-23e3e317b5ca-kube-api-access-xftgn\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-tpk56\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:06.597315 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.597283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:06.614523 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.614497    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:06.736906 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.736876    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"]
Jul 28 16:30:06.739876 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:30:06.739844    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff08653c_5659_4a8b_a721_f5484033bdc1.slice/crio-a90df06cb32ca17531fd53315506624402aeb1aa9b41d22eb027a9ab3444dfee WatchSource:0}: Error finding container a90df06cb32ca17531fd53315506624402aeb1aa9b41d22eb027a9ab3444dfee: Status 404 returned error can't find the container with id a90df06cb32ca17531fd53315506624402aeb1aa9b41d22eb027a9ab3444dfee
Jul 28 16:30:06.760180 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.760159    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"]
Jul 28 16:30:06.767947 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:30:06.767921    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a79e67_dc23_45ac_a629_23e3e317b5ca.slice/crio-5dda2656bb85f8c8cfd74d4fa19ef9dddbcc1c1ed7b8dcee3f4e43a9e06be143 WatchSource:0}: Error finding container 5dda2656bb85f8c8cfd74d4fa19ef9dddbcc1c1ed7b8dcee3f4e43a9e06be143: Status 404 returned error can't find the container with id 5dda2656bb85f8c8cfd74d4fa19ef9dddbcc1c1ed7b8dcee3f4e43a9e06be143
Jul 28 16:30:06.950070 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.950037    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" event={"ID":"ff08653c-5659-4a8b-a721-f5484033bdc1","Type":"ContainerStarted","Data":"3a53356b32078b3dc5f96411110796091aa8610f00492c42bc48bee5337d1270"}
Jul 28 16:30:06.950254 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.950103    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" event={"ID":"ff08653c-5659-4a8b-a721-f5484033bdc1","Type":"ContainerStarted","Data":"a90df06cb32ca17531fd53315506624402aeb1aa9b41d22eb027a9ab3444dfee"}
Jul 28 16:30:06.952772 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.952725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" event={"ID":"02a79e67-dc23-45ac-a629-23e3e317b5ca","Type":"ContainerStarted","Data":"f3cdb6968c5fb7f5a780d0494d515b3f0a577e9caf7487d754aeabdf1444d288"}
Jul 28 16:30:06.952866 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:06.952799    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" event={"ID":"02a79e67-dc23-45ac-a629-23e3e317b5ca","Type":"ContainerStarted","Data":"5dda2656bb85f8c8cfd74d4fa19ef9dddbcc1c1ed7b8dcee3f4e43a9e06be143"}
Jul 28 16:30:07.981633 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:07.981600    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3" path="/var/lib/kubelet/pods/b8dbc806-31ac-41b9-82f6-eb0a3e54ddc3/volumes"
Jul 28 16:30:10.972049 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:10.972002    2581 generic.go:356] "Generic (PLEG): container finished" podID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerID="3a53356b32078b3dc5f96411110796091aa8610f00492c42bc48bee5337d1270" exitCode=0
Jul 28 16:30:10.972426 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:10.972079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" event={"ID":"ff08653c-5659-4a8b-a721-f5484033bdc1","Type":"ContainerDied","Data":"3a53356b32078b3dc5f96411110796091aa8610f00492c42bc48bee5337d1270"}
Jul 28 16:30:10.973690 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:10.973664    2581 generic.go:356] "Generic (PLEG): container finished" podID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerID="f3cdb6968c5fb7f5a780d0494d515b3f0a577e9caf7487d754aeabdf1444d288" exitCode=0
Jul 28 16:30:10.973825 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:10.973693    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" event={"ID":"02a79e67-dc23-45ac-a629-23e3e317b5ca","Type":"ContainerDied","Data":"f3cdb6968c5fb7f5a780d0494d515b3f0a577e9caf7487d754aeabdf1444d288"}
Jul 28 16:30:11.981166 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:11.981128    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" event={"ID":"ff08653c-5659-4a8b-a721-f5484033bdc1","Type":"ContainerStarted","Data":"dacbf0f44e8b18370a1cdf8b007acfa80097cf5bf726c26726297bbdbfa1d506"}
Jul 28 16:30:11.981611 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:11.981191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" event={"ID":"ff08653c-5659-4a8b-a721-f5484033bdc1","Type":"ContainerStarted","Data":"d086c80e7524701e89391d45b4c75a13c7de390b57618ad33aaa625aa19c0612"}
Jul 28 16:30:11.981611 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:11.981397    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:11.983108 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:11.983062    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" event={"ID":"02a79e67-dc23-45ac-a629-23e3e317b5ca","Type":"ContainerStarted","Data":"171b4caca259be5a24eddb40a39bffbbc0abb06494252f0edc1b0459d097b96e"}
Jul 28 16:30:11.983221 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:11.983136    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" event={"ID":"02a79e67-dc23-45ac-a629-23e3e317b5ca","Type":"ContainerStarted","Data":"3ea9cb520add368055d58bd484025ff49f3520ab57cd9608c4231e7d9cd2eada"}
Jul 28 16:30:11.983333 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:11.983316    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:12.002205 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:12.002155    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" podStartSLOduration=2.35924162 podStartE2EDuration="6.00213958s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.64289796s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:30:12.001148818 +0000 UTC m=+1698.559705382" watchObservedRunningTime="2026-07-28 16:30:12.00213958 +0000 UTC m=+1698.560696124"
Jul 28 16:30:12.022646 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:12.022607    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" podStartSLOduration=2.173670124 podStartE2EDuration="6.022595459s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.848925335s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:30:12.021329619 +0000 UTC m=+1698.579886174" watchObservedRunningTime="2026-07-28 16:30:12.022595459 +0000 UTC m=+1698.581152000"
Jul 28 16:30:12.988671 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:12.988639    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:12.989055 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:12.988692    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:12.990066 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:12.990040    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 28 16:30:12.990178 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:12.990089    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 28 16:30:13.992361 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:13.992316    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 28 16:30:13.992793 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:13.992441    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 28 16:30:18.997612 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:18.997537    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:30:18.997945 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:18.997628    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:30:18.998149 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:18.998121    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 28 16:30:18.998197 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:18.998179    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 28 16:30:28.998687 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:28.998644    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 28 16:30:28.999158 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:28.998644    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 28 16:30:38.998212 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:38.998174    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 28 16:30:38.998212 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:38.998191    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 28 16:30:48.998345 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:48.998302    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 28 16:30:48.998721 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:48.998302    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 28 16:30:58.998702 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:58.998662    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 28 16:30:58.999078 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:30:58.998661    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 28 16:31:08.998811 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:08.998773    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 28 16:31:08.999284 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:08.998774    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 28 16:31:18.998510 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:18.998471    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 28 16:31:18.998956 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:18.998472    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 28 16:31:25.981418 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:25.981390    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:31:28.999175 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:28.999147    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:31:36.194569 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:36.194525    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"]
Jul 28 16:31:36.195086 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:36.194849    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="kserve-container" containerID="cri-o://d086c80e7524701e89391d45b4c75a13c7de390b57618ad33aaa625aa19c0612" gracePeriod=30
Jul 28 16:31:36.195086 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:36.195055    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="kube-rbac-proxy" containerID="cri-o://dacbf0f44e8b18370a1cdf8b007acfa80097cf5bf726c26726297bbdbfa1d506" gracePeriod=30
Jul 28 16:31:36.233908 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:36.233882    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"]
Jul 28 16:31:36.234149 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:36.234126    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kserve-container" containerID="cri-o://3ea9cb520add368055d58bd484025ff49f3520ab57cd9608c4231e7d9cd2eada" gracePeriod=30
Jul 28 16:31:36.234231 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:36.234164    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kube-rbac-proxy" containerID="cri-o://171b4caca259be5a24eddb40a39bffbbc0abb06494252f0edc1b0459d097b96e" gracePeriod=30
Jul 28 16:31:37.296368 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:37.296335    2581 generic.go:356] "Generic (PLEG): container finished" podID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerID="dacbf0f44e8b18370a1cdf8b007acfa80097cf5bf726c26726297bbdbfa1d506" exitCode=2
Jul 28 16:31:37.296809 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:37.296402    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" event={"ID":"ff08653c-5659-4a8b-a721-f5484033bdc1","Type":"ContainerDied","Data":"dacbf0f44e8b18370a1cdf8b007acfa80097cf5bf726c26726297bbdbfa1d506"}
Jul 28 16:31:37.298305 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:37.298280    2581 generic.go:356] "Generic (PLEG): container finished" podID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerID="171b4caca259be5a24eddb40a39bffbbc0abb06494252f0edc1b0459d097b96e" exitCode=2
Jul 28 16:31:37.298408 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:37.298353    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" event={"ID":"02a79e67-dc23-45ac-a629-23e3e317b5ca","Type":"ContainerDied","Data":"171b4caca259be5a24eddb40a39bffbbc0abb06494252f0edc1b0459d097b96e"}
Jul 28 16:31:38.992711 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:38.992665    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 28 16:31:38.993126 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:38.992666    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 28 16:31:38.999047 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:38.999026    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 28 16:31:41.063302 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.063278    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:31:41.093907 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.093881    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:31:41.098338 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.098279    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff08653c-5659-4a8b-a721-f5484033bdc1-proxy-tls\") pod \"ff08653c-5659-4a8b-a721-f5484033bdc1\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") "
Jul 28 16:31:41.098338 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.098310    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff08653c-5659-4a8b-a721-f5484033bdc1-cabundle-cert\") pod \"ff08653c-5659-4a8b-a721-f5484033bdc1\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") "
Jul 28 16:31:41.098520 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.098422    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff08653c-5659-4a8b-a721-f5484033bdc1-kube-rbac-proxy-sar-config\") pod \"ff08653c-5659-4a8b-a721-f5484033bdc1\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") "
Jul 28 16:31:41.098520 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.098466    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff08653c-5659-4a8b-a721-f5484033bdc1-kserve-provision-location\") pod \"ff08653c-5659-4a8b-a721-f5484033bdc1\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") "
Jul 28 16:31:41.098520 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.098502    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5pzl\" (UniqueName: \"kubernetes.io/projected/ff08653c-5659-4a8b-a721-f5484033bdc1-kube-api-access-f5pzl\") pod \"ff08653c-5659-4a8b-a721-f5484033bdc1\" (UID: \"ff08653c-5659-4a8b-a721-f5484033bdc1\") "
Jul 28 16:31:41.098731 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.098702    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff08653c-5659-4a8b-a721-f5484033bdc1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff08653c-5659-4a8b-a721-f5484033bdc1" (UID: "ff08653c-5659-4a8b-a721-f5484033bdc1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:31:41.098814 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.098768    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff08653c-5659-4a8b-a721-f5484033bdc1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ff08653c-5659-4a8b-a721-f5484033bdc1" (UID: "ff08653c-5659-4a8b-a721-f5484033bdc1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:31:41.099039 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.099002    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff08653c-5659-4a8b-a721-f5484033bdc1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff08653c-5659-4a8b-a721-f5484033bdc1" (UID: "ff08653c-5659-4a8b-a721-f5484033bdc1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:31:41.100597 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.100566    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff08653c-5659-4a8b-a721-f5484033bdc1-kube-api-access-f5pzl" (OuterVolumeSpecName: "kube-api-access-f5pzl") pod "ff08653c-5659-4a8b-a721-f5484033bdc1" (UID: "ff08653c-5659-4a8b-a721-f5484033bdc1"). InnerVolumeSpecName "kube-api-access-f5pzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:31:41.100696 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.100669    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff08653c-5659-4a8b-a721-f5484033bdc1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff08653c-5659-4a8b-a721-f5484033bdc1" (UID: "ff08653c-5659-4a8b-a721-f5484033bdc1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:31:41.199135 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.199101    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xftgn\" (UniqueName: \"kubernetes.io/projected/02a79e67-dc23-45ac-a629-23e3e317b5ca-kube-api-access-xftgn\") pod \"02a79e67-dc23-45ac-a629-23e3e317b5ca\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") "
Jul 28 16:31:41.199299 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.199150    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02a79e67-dc23-45ac-a629-23e3e317b5ca-kserve-provision-location\") pod \"02a79e67-dc23-45ac-a629-23e3e317b5ca\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") "
Jul 28 16:31:41.199299 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.199195    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a79e67-dc23-45ac-a629-23e3e317b5ca-proxy-tls\") pod \"02a79e67-dc23-45ac-a629-23e3e317b5ca\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") "
Jul 28 16:31:41.199299 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.199222    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a79e67-dc23-45ac-a629-23e3e317b5ca-kube-rbac-proxy-sar-config\") pod \"02a79e67-dc23-45ac-a629-23e3e317b5ca\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") "
Jul 28 16:31:41.199299 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.199253    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02a79e67-dc23-45ac-a629-23e3e317b5ca-cabundle-cert\") pod \"02a79e67-dc23-45ac-a629-23e3e317b5ca\" (UID: \"02a79e67-dc23-45ac-a629-23e3e317b5ca\") "
Jul 28 16:31:41.199499 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.199466    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff08653c-5659-4a8b-a721-f5484033bdc1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:31:41.199499 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.199484    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff08653c-5659-4a8b-a721-f5484033bdc1-kserve-provision-location\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:31:41.199607 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.199502    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5pzl\" (UniqueName: \"kubernetes.io/projected/ff08653c-5659-4a8b-a721-f5484033bdc1-kube-api-access-f5pzl\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:31:41.199607 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.199515    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff08653c-5659-4a8b-a721-f5484033bdc1-proxy-tls\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:31:41.199607 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.199530    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff08653c-5659-4a8b-a721-f5484033bdc1-cabundle-cert\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:31:41.199607 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.199562    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a79e67-dc23-45ac-a629-23e3e317b5ca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "02a79e67-dc23-45ac-a629-23e3e317b5ca" (UID: "02a79e67-dc23-45ac-a629-23e3e317b5ca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:31:41.199607 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.199579    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a79e67-dc23-45ac-a629-23e3e317b5ca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "02a79e67-dc23-45ac-a629-23e3e317b5ca" (UID: "02a79e67-dc23-45ac-a629-23e3e317b5ca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:31:41.199607 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.199606    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a79e67-dc23-45ac-a629-23e3e317b5ca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "02a79e67-dc23-45ac-a629-23e3e317b5ca" (UID: "02a79e67-dc23-45ac-a629-23e3e317b5ca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:31:41.201188 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.201157    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a79e67-dc23-45ac-a629-23e3e317b5ca-kube-api-access-xftgn" (OuterVolumeSpecName: "kube-api-access-xftgn") pod "02a79e67-dc23-45ac-a629-23e3e317b5ca" (UID: "02a79e67-dc23-45ac-a629-23e3e317b5ca"). InnerVolumeSpecName "kube-api-access-xftgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:31:41.201244 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.201194    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a79e67-dc23-45ac-a629-23e3e317b5ca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "02a79e67-dc23-45ac-a629-23e3e317b5ca" (UID: "02a79e67-dc23-45ac-a629-23e3e317b5ca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:31:41.299832 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.299799    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02a79e67-dc23-45ac-a629-23e3e317b5ca-cabundle-cert\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:31:41.299832 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.299825    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xftgn\" (UniqueName: \"kubernetes.io/projected/02a79e67-dc23-45ac-a629-23e3e317b5ca-kube-api-access-xftgn\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:31:41.299832 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.299835    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02a79e67-dc23-45ac-a629-23e3e317b5ca-kserve-provision-location\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:31:41.300057 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.299843    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a79e67-dc23-45ac-a629-23e3e317b5ca-proxy-tls\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:31:41.300057 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.299852    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a79e67-dc23-45ac-a629-23e3e317b5ca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:31:41.314434 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.314400    2581 generic.go:356] "Generic (PLEG): container finished" podID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerID="d086c80e7524701e89391d45b4c75a13c7de390b57618ad33aaa625aa19c0612" exitCode=0
Jul 28 16:31:41.314573 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.314458    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"
Jul 28 16:31:41.314573 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.314483    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" event={"ID":"ff08653c-5659-4a8b-a721-f5484033bdc1","Type":"ContainerDied","Data":"d086c80e7524701e89391d45b4c75a13c7de390b57618ad33aaa625aa19c0612"}
Jul 28 16:31:41.314573 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.314546    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq" event={"ID":"ff08653c-5659-4a8b-a721-f5484033bdc1","Type":"ContainerDied","Data":"a90df06cb32ca17531fd53315506624402aeb1aa9b41d22eb027a9ab3444dfee"}
Jul 28 16:31:41.314728 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.314580    2581 scope.go:117] "RemoveContainer" containerID="dacbf0f44e8b18370a1cdf8b007acfa80097cf5bf726c26726297bbdbfa1d506"
Jul 28 16:31:41.316450 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.316430    2581 generic.go:356] "Generic (PLEG): container finished" podID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerID="3ea9cb520add368055d58bd484025ff49f3520ab57cd9608c4231e7d9cd2eada" exitCode=0
Jul 28 16:31:41.316537 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.316487    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"
Jul 28 16:31:41.316537 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.316515    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" event={"ID":"02a79e67-dc23-45ac-a629-23e3e317b5ca","Type":"ContainerDied","Data":"3ea9cb520add368055d58bd484025ff49f3520ab57cd9608c4231e7d9cd2eada"}
Jul 28 16:31:41.316654 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.316577    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56" event={"ID":"02a79e67-dc23-45ac-a629-23e3e317b5ca","Type":"ContainerDied","Data":"5dda2656bb85f8c8cfd74d4fa19ef9dddbcc1c1ed7b8dcee3f4e43a9e06be143"}
Jul 28 16:31:41.322823 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.322790    2581 scope.go:117] "RemoveContainer" containerID="d086c80e7524701e89391d45b4c75a13c7de390b57618ad33aaa625aa19c0612"
Jul 28 16:31:41.330610 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.330596    2581 scope.go:117] "RemoveContainer" containerID="3a53356b32078b3dc5f96411110796091aa8610f00492c42bc48bee5337d1270"
Jul 28 16:31:41.337506 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.337491    2581 scope.go:117] "RemoveContainer" containerID="dacbf0f44e8b18370a1cdf8b007acfa80097cf5bf726c26726297bbdbfa1d506"
Jul 28 16:31:41.337776 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:31:41.337755    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dacbf0f44e8b18370a1cdf8b007acfa80097cf5bf726c26726297bbdbfa1d506\": container with ID starting with dacbf0f44e8b18370a1cdf8b007acfa80097cf5bf726c26726297bbdbfa1d506 not found: ID does not exist" containerID="dacbf0f44e8b18370a1cdf8b007acfa80097cf5bf726c26726297bbdbfa1d506"
Jul 28 16:31:41.337776 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.337780    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dacbf0f44e8b18370a1cdf8b007acfa80097cf5bf726c26726297bbdbfa1d506"} err="failed to get container status \"dacbf0f44e8b18370a1cdf8b007acfa80097cf5bf726c26726297bbdbfa1d506\": rpc error: code = NotFound desc = could not find container \"dacbf0f44e8b18370a1cdf8b007acfa80097cf5bf726c26726297bbdbfa1d506\": container with ID starting with dacbf0f44e8b18370a1cdf8b007acfa80097cf5bf726c26726297bbdbfa1d506 not found: ID does not exist"
Jul 28 16:31:41.338282 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.337799    2581 scope.go:117] "RemoveContainer" containerID="d086c80e7524701e89391d45b4c75a13c7de390b57618ad33aaa625aa19c0612"
Jul 28 16:31:41.338282 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:31:41.338092    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d086c80e7524701e89391d45b4c75a13c7de390b57618ad33aaa625aa19c0612\": container with ID starting with d086c80e7524701e89391d45b4c75a13c7de390b57618ad33aaa625aa19c0612 not found: ID does not exist" containerID="d086c80e7524701e89391d45b4c75a13c7de390b57618ad33aaa625aa19c0612"
Jul 28 16:31:41.338282 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.338132    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d086c80e7524701e89391d45b4c75a13c7de390b57618ad33aaa625aa19c0612"} err="failed to get container status \"d086c80e7524701e89391d45b4c75a13c7de390b57618ad33aaa625aa19c0612\": rpc error: code = NotFound desc = could not find container \"d086c80e7524701e89391d45b4c75a13c7de390b57618ad33aaa625aa19c0612\": container with ID starting with d086c80e7524701e89391d45b4c75a13c7de390b57618ad33aaa625aa19c0612 not found: ID does not exist"
Jul 28 16:31:41.338282 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.338166    2581 scope.go:117] "RemoveContainer" containerID="3a53356b32078b3dc5f96411110796091aa8610f00492c42bc48bee5337d1270"
Jul 28 16:31:41.338488 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:31:41.338462    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a53356b32078b3dc5f96411110796091aa8610f00492c42bc48bee5337d1270\": container with ID starting with 3a53356b32078b3dc5f96411110796091aa8610f00492c42bc48bee5337d1270 not found: ID does not exist" containerID="3a53356b32078b3dc5f96411110796091aa8610f00492c42bc48bee5337d1270"
Jul 28 16:31:41.338535 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.338494    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a53356b32078b3dc5f96411110796091aa8610f00492c42bc48bee5337d1270"} err="failed to get container status \"3a53356b32078b3dc5f96411110796091aa8610f00492c42bc48bee5337d1270\": rpc error: code = NotFound desc = could not find container \"3a53356b32078b3dc5f96411110796091aa8610f00492c42bc48bee5337d1270\": container with ID starting with 3a53356b32078b3dc5f96411110796091aa8610f00492c42bc48bee5337d1270 not found: ID does not exist"
Jul 28 16:31:41.338535 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.338515    2581 scope.go:117] "RemoveContainer" containerID="171b4caca259be5a24eddb40a39bffbbc0abb06494252f0edc1b0459d097b96e"
Jul 28 16:31:41.339883 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.339867    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"]
Jul 28 16:31:41.345086 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.345071    2581 scope.go:117] "RemoveContainer" containerID="3ea9cb520add368055d58bd484025ff49f3520ab57cd9608c4231e7d9cd2eada"
Jul 28 16:31:41.348588 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.348535    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-pl6gq"]
Jul 28 16:31:41.352187 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.352169    2581 scope.go:117] "RemoveContainer" containerID="f3cdb6968c5fb7f5a780d0494d515b3f0a577e9caf7487d754aeabdf1444d288"
Jul 28 16:31:41.358727 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.358712    2581 scope.go:117] "RemoveContainer" containerID="171b4caca259be5a24eddb40a39bffbbc0abb06494252f0edc1b0459d097b96e"
Jul 28 16:31:41.358950 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:31:41.358932    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"171b4caca259be5a24eddb40a39bffbbc0abb06494252f0edc1b0459d097b96e\": container with ID starting with 171b4caca259be5a24eddb40a39bffbbc0abb06494252f0edc1b0459d097b96e not found: ID does not exist" containerID="171b4caca259be5a24eddb40a39bffbbc0abb06494252f0edc1b0459d097b96e"
Jul 28 16:31:41.358991 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.358956    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"171b4caca259be5a24eddb40a39bffbbc0abb06494252f0edc1b0459d097b96e"} err="failed to get container status \"171b4caca259be5a24eddb40a39bffbbc0abb06494252f0edc1b0459d097b96e\": rpc error: code = NotFound desc = could not find container \"171b4caca259be5a24eddb40a39bffbbc0abb06494252f0edc1b0459d097b96e\": container with ID starting with 171b4caca259be5a24eddb40a39bffbbc0abb06494252f0edc1b0459d097b96e not found: ID does not exist"
Jul 28 16:31:41.358991 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.358978    2581 scope.go:117] "RemoveContainer" containerID="3ea9cb520add368055d58bd484025ff49f3520ab57cd9608c4231e7d9cd2eada"
Jul 28 16:31:41.359236 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:31:41.359218    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea9cb520add368055d58bd484025ff49f3520ab57cd9608c4231e7d9cd2eada\": container with ID starting with 3ea9cb520add368055d58bd484025ff49f3520ab57cd9608c4231e7d9cd2eada not found: ID does not exist" containerID="3ea9cb520add368055d58bd484025ff49f3520ab57cd9608c4231e7d9cd2eada"
Jul 28 16:31:41.359282 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.359244    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea9cb520add368055d58bd484025ff49f3520ab57cd9608c4231e7d9cd2eada"} err="failed to get container status \"3ea9cb520add368055d58bd484025ff49f3520ab57cd9608c4231e7d9cd2eada\": rpc error: code = NotFound desc = could not find container \"3ea9cb520add368055d58bd484025ff49f3520ab57cd9608c4231e7d9cd2eada\": container with ID starting with 3ea9cb520add368055d58bd484025ff49f3520ab57cd9608c4231e7d9cd2eada not found: ID does not exist"
Jul 28 16:31:41.359282 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.359259    2581 scope.go:117] "RemoveContainer" containerID="f3cdb6968c5fb7f5a780d0494d515b3f0a577e9caf7487d754aeabdf1444d288"
Jul 28 16:31:41.359507 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:31:41.359489    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3cdb6968c5fb7f5a780d0494d515b3f0a577e9caf7487d754aeabdf1444d288\": container with ID starting with f3cdb6968c5fb7f5a780d0494d515b3f0a577e9caf7487d754aeabdf1444d288 not found: ID does not exist" containerID="f3cdb6968c5fb7f5a780d0494d515b3f0a577e9caf7487d754aeabdf1444d288"
Jul 28 16:31:41.359556 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.359511    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3cdb6968c5fb7f5a780d0494d515b3f0a577e9caf7487d754aeabdf1444d288"} err="failed to get container status \"f3cdb6968c5fb7f5a780d0494d515b3f0a577e9caf7487d754aeabdf1444d288\": rpc error: code = NotFound desc = could not find container \"f3cdb6968c5fb7f5a780d0494d515b3f0a577e9caf7487d754aeabdf1444d288\": container with ID starting with f3cdb6968c5fb7f5a780d0494d515b3f0a577e9caf7487d754aeabdf1444d288 not found: ID does not exist"
Jul 28 16:31:41.367724 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.367703    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"]
Jul 28 16:31:41.374544 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.374525    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-tpk56"]
Jul 28 16:31:41.982073 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.982039    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" path="/var/lib/kubelet/pods/02a79e67-dc23-45ac-a629-23e3e317b5ca/volumes"
Jul 28 16:31:41.982523 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:31:41.982510    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" path="/var/lib/kubelet/pods/ff08653c-5659-4a8b-a721-f5484033bdc1/volumes"
Jul 28 16:33:31.804185 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.804151    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"]
Jul 28 16:33:31.804659 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.804431    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="storage-initializer"
Jul 28 16:33:31.804659 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.804444    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="storage-initializer"
Jul 28 16:33:31.804659 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.804451    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kserve-container"
Jul 28 16:33:31.804659 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.804457    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kserve-container"
Jul 28 16:33:31.804659 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.804464    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="kube-rbac-proxy"
Jul 28 16:33:31.804659 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.804470    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="kube-rbac-proxy"
Jul 28 16:33:31.804659 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.804476    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="storage-initializer"
Jul 28 16:33:31.804659 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.804484    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="storage-initializer"
Jul 28 16:33:31.804659 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.804495    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kube-rbac-proxy"
Jul 28 16:33:31.804659 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.804501    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kube-rbac-proxy"
Jul 28 16:33:31.804659 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.804509    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="kserve-container"
Jul 28 16:33:31.804659 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.804514    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="kserve-container"
Jul 28 16:33:31.804659 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.804554    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kube-rbac-proxy"
Jul 28 16:33:31.804659 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.804563    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="kserve-container"
Jul 28 16:33:31.804659 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.804571    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff08653c-5659-4a8b-a721-f5484033bdc1" containerName="kube-rbac-proxy"
Jul 28 16:33:31.804659 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.804580    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="02a79e67-dc23-45ac-a629-23e3e317b5ca" containerName="kserve-container"
Jul 28 16:33:31.807663 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.807645    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:31.809746 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.809724    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-5198af-kube-rbac-proxy-sar-config\""
Jul 28 16:33:31.809860 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.809733    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-5198af\""
Jul 28 16:33:31.809925 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.809894    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-5198af-dockercfg-8b5x6\""
Jul 28 16:33:31.810011 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.809987    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:33:31.810419 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.810401    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:33:31.810528 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.810436    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:33:31.810528 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.810434    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-5198af-predictor-serving-cert\""
Jul 28 16:33:31.818755 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.818732    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"]
Jul 28 16:33:31.877873 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.877834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-proxy-tls\") pod \"isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") " pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:31.878078 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.877889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") " pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:31.878078 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.877976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdw2t\" (UniqueName: \"kubernetes.io/projected/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-kube-api-access-mdw2t\") pod \"isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") " pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:31.878078 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.878006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-cabundle-cert\") pod \"isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") " pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:31.878198 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.878097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-kserve-provision-location\") pod \"isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") " pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:31.979029 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.978977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-proxy-tls\") pod \"isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") " pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:31.979199 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.979060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") " pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:31.979199 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.979110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdw2t\" (UniqueName: \"kubernetes.io/projected/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-kube-api-access-mdw2t\") pod \"isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") " pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:31.979199 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.979136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-cabundle-cert\") pod \"isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") " pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:31.979199 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.979176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-kserve-provision-location\") pod \"isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") " pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:31.979646 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.979622    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-kserve-provision-location\") pod \"isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") " pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:31.979965 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.979930    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-cabundle-cert\") pod \"isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") " pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:31.980059 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.979930    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") " pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:31.981759 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.981738    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-proxy-tls\") pod \"isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") " pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:31.990138 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:31.990116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdw2t\" (UniqueName: \"kubernetes.io/projected/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-kube-api-access-mdw2t\") pod \"isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") " pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:32.118512 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:32.118418    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:32.250788 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:32.250766    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"]
Jul 28 16:33:32.253480 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:33:32.253456    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207b6fd9_daf0_4ee7_81cb_810f9fb42f68.slice/crio-24d120c5a58faef81464513d3d156a47c12b8a13f57a23a4d81eeb26661246d6 WatchSource:0}: Error finding container 24d120c5a58faef81464513d3d156a47c12b8a13f57a23a4d81eeb26661246d6: Status 404 returned error can't find the container with id 24d120c5a58faef81464513d3d156a47c12b8a13f57a23a4d81eeb26661246d6
Jul 28 16:33:32.698195 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:32.698157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl" event={"ID":"207b6fd9-daf0-4ee7-81cb-810f9fb42f68","Type":"ContainerStarted","Data":"cd4f72d89a35fce5e83271c803621975a564273d7804ea3df5388549272571ba"}
Jul 28 16:33:32.698395 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:32.698231    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl" event={"ID":"207b6fd9-daf0-4ee7-81cb-810f9fb42f68","Type":"ContainerStarted","Data":"24d120c5a58faef81464513d3d156a47c12b8a13f57a23a4d81eeb26661246d6"}
Jul 28 16:33:37.715943 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:37.715913    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl_207b6fd9-daf0-4ee7-81cb-810f9fb42f68/storage-initializer/0.log"
Jul 28 16:33:37.716328 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:37.715954    2581 generic.go:356] "Generic (PLEG): container finished" podID="207b6fd9-daf0-4ee7-81cb-810f9fb42f68" containerID="cd4f72d89a35fce5e83271c803621975a564273d7804ea3df5388549272571ba" exitCode=1
Jul 28 16:33:37.716328 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:37.716003    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl" event={"ID":"207b6fd9-daf0-4ee7-81cb-810f9fb42f68","Type":"ContainerDied","Data":"cd4f72d89a35fce5e83271c803621975a564273d7804ea3df5388549272571ba"}
Jul 28 16:33:38.720498 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:38.720472    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl_207b6fd9-daf0-4ee7-81cb-810f9fb42f68/storage-initializer/0.log"
Jul 28 16:33:38.720918 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:38.720565    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl" event={"ID":"207b6fd9-daf0-4ee7-81cb-810f9fb42f68","Type":"ContainerStarted","Data":"2762d0d1ea1fe418514d03e82773233c3e4fc6e5009b8494187e89093266bbb4"}
Jul 28 16:33:43.740353 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:43.740323    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl_207b6fd9-daf0-4ee7-81cb-810f9fb42f68/storage-initializer/1.log"
Jul 28 16:33:43.740804 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:43.740745    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl_207b6fd9-daf0-4ee7-81cb-810f9fb42f68/storage-initializer/0.log"
Jul 28 16:33:43.740804 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:43.740784    2581 generic.go:356] "Generic (PLEG): container finished" podID="207b6fd9-daf0-4ee7-81cb-810f9fb42f68" containerID="2762d0d1ea1fe418514d03e82773233c3e4fc6e5009b8494187e89093266bbb4" exitCode=1
Jul 28 16:33:43.740992 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:43.740818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl" event={"ID":"207b6fd9-daf0-4ee7-81cb-810f9fb42f68","Type":"ContainerDied","Data":"2762d0d1ea1fe418514d03e82773233c3e4fc6e5009b8494187e89093266bbb4"}
Jul 28 16:33:43.740992 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:43.740895    2581 scope.go:117] "RemoveContainer" containerID="cd4f72d89a35fce5e83271c803621975a564273d7804ea3df5388549272571ba"
Jul 28 16:33:43.741332 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:43.741316    2581 scope.go:117] "RemoveContainer" containerID="cd4f72d89a35fce5e83271c803621975a564273d7804ea3df5388549272571ba"
Jul 28 16:33:43.751149 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:33:43.751120    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl_kserve-ci-e2e-test(207b6fd9-daf0-4ee7-81cb-810f9fb42f68)\"" pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl" podUID="207b6fd9-daf0-4ee7-81cb-810f9fb42f68"
Jul 28 16:33:43.751323 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:33:43.751295    2581 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl_kserve-ci-e2e-test_207b6fd9-daf0-4ee7-81cb-810f9fb42f68_0 in pod sandbox 24d120c5a58faef81464513d3d156a47c12b8a13f57a23a4d81eeb26661246d6 from index: no such id: 'cd4f72d89a35fce5e83271c803621975a564273d7804ea3df5388549272571ba'" containerID="cd4f72d89a35fce5e83271c803621975a564273d7804ea3df5388549272571ba"
Jul 28 16:33:43.751372 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:43.751339    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4f72d89a35fce5e83271c803621975a564273d7804ea3df5388549272571ba"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl_kserve-ci-e2e-test_207b6fd9-daf0-4ee7-81cb-810f9fb42f68_0 in pod sandbox 24d120c5a58faef81464513d3d156a47c12b8a13f57a23a4d81eeb26661246d6 from index: no such id: 'cd4f72d89a35fce5e83271c803621975a564273d7804ea3df5388549272571ba'"
Jul 28 16:33:44.745870 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:44.745840    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl_207b6fd9-daf0-4ee7-81cb-810f9fb42f68/storage-initializer/1.log"
Jul 28 16:33:49.858841 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:49.858812    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"]
Jul 28 16:33:49.997055 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:49.997035    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl_207b6fd9-daf0-4ee7-81cb-810f9fb42f68/storage-initializer/1.log"
Jul 28 16:33:50.000675 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.000659    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:50.108256 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.108217    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-kserve-provision-location\") pod \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") "
Jul 28 16:33:50.108415 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.108269    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-kube-rbac-proxy-sar-config\") pod \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") "
Jul 28 16:33:50.108415 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.108301    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdw2t\" (UniqueName: \"kubernetes.io/projected/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-kube-api-access-mdw2t\") pod \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") "
Jul 28 16:33:50.108415 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.108324    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-cabundle-cert\") pod \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") "
Jul 28 16:33:50.108415 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.108350    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-proxy-tls\") pod \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\" (UID: \"207b6fd9-daf0-4ee7-81cb-810f9fb42f68\") "
Jul 28 16:33:50.108610 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.108517    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "207b6fd9-daf0-4ee7-81cb-810f9fb42f68" (UID: "207b6fd9-daf0-4ee7-81cb-810f9fb42f68"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:33:50.108687 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.108658    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "207b6fd9-daf0-4ee7-81cb-810f9fb42f68" (UID: "207b6fd9-daf0-4ee7-81cb-810f9fb42f68"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:33:50.108754 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.108663    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "207b6fd9-daf0-4ee7-81cb-810f9fb42f68" (UID: "207b6fd9-daf0-4ee7-81cb-810f9fb42f68"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:33:50.110336 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.110288    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-kube-api-access-mdw2t" (OuterVolumeSpecName: "kube-api-access-mdw2t") pod "207b6fd9-daf0-4ee7-81cb-810f9fb42f68" (UID: "207b6fd9-daf0-4ee7-81cb-810f9fb42f68"). InnerVolumeSpecName "kube-api-access-mdw2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:33:50.110423 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.110394    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "207b6fd9-daf0-4ee7-81cb-810f9fb42f68" (UID: "207b6fd9-daf0-4ee7-81cb-810f9fb42f68"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:33:50.209325 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.209300    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-kserve-provision-location\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:33:50.209325 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.209321    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:33:50.209459 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.209331    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdw2t\" (UniqueName: \"kubernetes.io/projected/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-kube-api-access-mdw2t\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:33:50.209459 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.209339    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-cabundle-cert\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:33:50.209459 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.209348    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/207b6fd9-daf0-4ee7-81cb-810f9fb42f68-proxy-tls\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:33:50.766340 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.766316    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl_207b6fd9-daf0-4ee7-81cb-810f9fb42f68/storage-initializer/1.log"
Jul 28 16:33:50.766500 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.766396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl" event={"ID":"207b6fd9-daf0-4ee7-81cb-810f9fb42f68","Type":"ContainerDied","Data":"24d120c5a58faef81464513d3d156a47c12b8a13f57a23a4d81eeb26661246d6"}
Jul 28 16:33:50.766500 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.766409    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"
Jul 28 16:33:50.766500 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.766455    2581 scope.go:117] "RemoveContainer" containerID="2762d0d1ea1fe418514d03e82773233c3e4fc6e5009b8494187e89093266bbb4"
Jul 28 16:33:50.803101 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.803081    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"]
Jul 28 16:33:50.807599 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:50.807575    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5198af-predictor-7bb4bffff4-5jrgl"]
Jul 28 16:33:51.981207 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:51.981179    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="207b6fd9-daf0-4ee7-81cb-810f9fb42f68" path="/var/lib/kubelet/pods/207b6fd9-daf0-4ee7-81cb-810f9fb42f68/volumes"
Jul 28 16:33:55.083570 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.083538    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"]
Jul 28 16:33:55.083935 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.083816    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="207b6fd9-daf0-4ee7-81cb-810f9fb42f68" containerName="storage-initializer"
Jul 28 16:33:55.083935 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.083827    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="207b6fd9-daf0-4ee7-81cb-810f9fb42f68" containerName="storage-initializer"
Jul 28 16:33:55.083935 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.083873    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="207b6fd9-daf0-4ee7-81cb-810f9fb42f68" containerName="storage-initializer"
Jul 28 16:33:55.083935 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.083922    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="207b6fd9-daf0-4ee7-81cb-810f9fb42f68" containerName="storage-initializer"
Jul 28 16:33:55.083935 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.083927    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="207b6fd9-daf0-4ee7-81cb-810f9fb42f68" containerName="storage-initializer"
Jul 28 16:33:55.084134 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.083978    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="207b6fd9-daf0-4ee7-81cb-810f9fb42f68" containerName="storage-initializer"
Jul 28 16:33:55.088292 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.088274    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:33:55.090409 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.090384    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-136677\""
Jul 28 16:33:55.090649 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.090633    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:33:55.090742 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.090635    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-136677-predictor-serving-cert\""
Jul 28 16:33:55.091325 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.091310    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:33:55.091546 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.091531    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-136677-kube-rbac-proxy-sar-config\""
Jul 28 16:33:55.091611 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.091535    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-136677-dockercfg-zf8qm\""
Jul 28 16:33:55.091690 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.091674    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:33:55.099009 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.098990    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"]
Jul 28 16:33:55.242534 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.242505    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8xvc\" (UniqueName: \"kubernetes.io/projected/841ef677-3054-4950-8de6-29fac7a070eb-kube-api-access-m8xvc\") pod \"isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:33:55.242652 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.242545    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/841ef677-3054-4950-8de6-29fac7a070eb-proxy-tls\") pod \"isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:33:55.242652 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.242578    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/841ef677-3054-4950-8de6-29fac7a070eb-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:33:55.242737 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.242659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/841ef677-3054-4950-8de6-29fac7a070eb-cabundle-cert\") pod \"isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:33:55.242737 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.242698    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/841ef677-3054-4950-8de6-29fac7a070eb-kserve-provision-location\") pod \"isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:33:55.343826 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.343769    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/841ef677-3054-4950-8de6-29fac7a070eb-cabundle-cert\") pod \"isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:33:55.343826 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.343801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/841ef677-3054-4950-8de6-29fac7a070eb-kserve-provision-location\") pod \"isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:33:55.343976 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.343829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8xvc\" (UniqueName: \"kubernetes.io/projected/841ef677-3054-4950-8de6-29fac7a070eb-kube-api-access-m8xvc\") pod \"isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:33:55.343976 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.343857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/841ef677-3054-4950-8de6-29fac7a070eb-proxy-tls\") pod \"isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:33:55.343976 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.343891    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/841ef677-3054-4950-8de6-29fac7a070eb-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:33:55.344264 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.344246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/841ef677-3054-4950-8de6-29fac7a070eb-kserve-provision-location\") pod \"isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:33:55.344480 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.344462    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/841ef677-3054-4950-8de6-29fac7a070eb-cabundle-cert\") pod \"isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:33:55.344537 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.344516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/841ef677-3054-4950-8de6-29fac7a070eb-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:33:55.346369 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.346351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/841ef677-3054-4950-8de6-29fac7a070eb-proxy-tls\") pod \"isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:33:55.353096 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.353075    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8xvc\" (UniqueName: \"kubernetes.io/projected/841ef677-3054-4950-8de6-29fac7a070eb-kube-api-access-m8xvc\") pod \"isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") " pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:33:55.398053 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.398032    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:33:55.524001 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.523974    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"]
Jul 28 16:33:55.527221 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:33:55.527192    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841ef677_3054_4950_8de6_29fac7a070eb.slice/crio-eb6fc4abdd8a261ac754526973c692ddcd964139120f8682816b89eeb1e8e38a WatchSource:0}: Error finding container eb6fc4abdd8a261ac754526973c692ddcd964139120f8682816b89eeb1e8e38a: Status 404 returned error can't find the container with id eb6fc4abdd8a261ac754526973c692ddcd964139120f8682816b89eeb1e8e38a
Jul 28 16:33:55.785479 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.785447    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5" event={"ID":"841ef677-3054-4950-8de6-29fac7a070eb","Type":"ContainerStarted","Data":"f82ea09a98114f49f88d53d08d7919826fabf4bddc74d2ad7f215ff53c7c88eb"}
Jul 28 16:33:55.785656 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:55.785506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5" event={"ID":"841ef677-3054-4950-8de6-29fac7a070eb","Type":"ContainerStarted","Data":"eb6fc4abdd8a261ac754526973c692ddcd964139120f8682816b89eeb1e8e38a"}
Jul 28 16:33:58.799359 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:58.799331    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5_841ef677-3054-4950-8de6-29fac7a070eb/storage-initializer/0.log"
Jul 28 16:33:58.799729 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:58.799369    2581 generic.go:356] "Generic (PLEG): container finished" podID="841ef677-3054-4950-8de6-29fac7a070eb" containerID="f82ea09a98114f49f88d53d08d7919826fabf4bddc74d2ad7f215ff53c7c88eb" exitCode=1
Jul 28 16:33:58.799729 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:58.799449    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5" event={"ID":"841ef677-3054-4950-8de6-29fac7a070eb","Type":"ContainerDied","Data":"f82ea09a98114f49f88d53d08d7919826fabf4bddc74d2ad7f215ff53c7c88eb"}
Jul 28 16:33:59.808764 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:59.804820    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5_841ef677-3054-4950-8de6-29fac7a070eb/storage-initializer/0.log"
Jul 28 16:33:59.808764 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:33:59.804892    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5" event={"ID":"841ef677-3054-4950-8de6-29fac7a070eb","Type":"ContainerStarted","Data":"9eb1c5189975f5b0c253460b10317574a6dc79b1632401de7f22da95e79cab4e"}
Jul 28 16:34:00.126925 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:00.126850    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"]
Jul 28 16:34:00.808242 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:00.808197    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5" podUID="841ef677-3054-4950-8de6-29fac7a070eb" containerName="storage-initializer" containerID="cri-o://9eb1c5189975f5b0c253460b10317574a6dc79b1632401de7f22da95e79cab4e" gracePeriod=30
Jul 28 16:34:04.362518 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.362493    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5_841ef677-3054-4950-8de6-29fac7a070eb/storage-initializer/1.log"
Jul 28 16:34:04.362912 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.362895    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5_841ef677-3054-4950-8de6-29fac7a070eb/storage-initializer/0.log"
Jul 28 16:34:04.365269 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.365253    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:34:04.509789 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.509755    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/841ef677-3054-4950-8de6-29fac7a070eb-cabundle-cert\") pod \"841ef677-3054-4950-8de6-29fac7a070eb\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") "
Jul 28 16:34:04.509998 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.509819    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8xvc\" (UniqueName: \"kubernetes.io/projected/841ef677-3054-4950-8de6-29fac7a070eb-kube-api-access-m8xvc\") pod \"841ef677-3054-4950-8de6-29fac7a070eb\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") "
Jul 28 16:34:04.509998 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.509857    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/841ef677-3054-4950-8de6-29fac7a070eb-kube-rbac-proxy-sar-config\") pod \"841ef677-3054-4950-8de6-29fac7a070eb\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") "
Jul 28 16:34:04.509998 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.509876    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/841ef677-3054-4950-8de6-29fac7a070eb-kserve-provision-location\") pod \"841ef677-3054-4950-8de6-29fac7a070eb\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") "
Jul 28 16:34:04.509998 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.509933    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/841ef677-3054-4950-8de6-29fac7a070eb-proxy-tls\") pod \"841ef677-3054-4950-8de6-29fac7a070eb\" (UID: \"841ef677-3054-4950-8de6-29fac7a070eb\") "
Jul 28 16:34:04.510246 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.510215    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/841ef677-3054-4950-8de6-29fac7a070eb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "841ef677-3054-4950-8de6-29fac7a070eb" (UID: "841ef677-3054-4950-8de6-29fac7a070eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:34:04.510307 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.510235    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841ef677-3054-4950-8de6-29fac7a070eb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "841ef677-3054-4950-8de6-29fac7a070eb" (UID: "841ef677-3054-4950-8de6-29fac7a070eb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:34:04.510361 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.510324    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841ef677-3054-4950-8de6-29fac7a070eb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "841ef677-3054-4950-8de6-29fac7a070eb" (UID: "841ef677-3054-4950-8de6-29fac7a070eb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:34:04.511994 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.511973    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841ef677-3054-4950-8de6-29fac7a070eb-kube-api-access-m8xvc" (OuterVolumeSpecName: "kube-api-access-m8xvc") pod "841ef677-3054-4950-8de6-29fac7a070eb" (UID: "841ef677-3054-4950-8de6-29fac7a070eb"). InnerVolumeSpecName "kube-api-access-m8xvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:34:04.512068 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.512024    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/841ef677-3054-4950-8de6-29fac7a070eb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "841ef677-3054-4950-8de6-29fac7a070eb" (UID: "841ef677-3054-4950-8de6-29fac7a070eb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:34:04.611230 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.611192    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/841ef677-3054-4950-8de6-29fac7a070eb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:34:04.611230 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.611224    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/841ef677-3054-4950-8de6-29fac7a070eb-kserve-provision-location\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:34:04.611230 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.611235    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/841ef677-3054-4950-8de6-29fac7a070eb-proxy-tls\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:34:04.611467 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.611244    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/841ef677-3054-4950-8de6-29fac7a070eb-cabundle-cert\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:34:04.611467 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.611253    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8xvc\" (UniqueName: \"kubernetes.io/projected/841ef677-3054-4950-8de6-29fac7a070eb-kube-api-access-m8xvc\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:34:04.823803 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.823720    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5_841ef677-3054-4950-8de6-29fac7a070eb/storage-initializer/1.log"
Jul 28 16:34:04.824194 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.824178    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5_841ef677-3054-4950-8de6-29fac7a070eb/storage-initializer/0.log"
Jul 28 16:34:04.824275 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.824211    2581 generic.go:356] "Generic (PLEG): container finished" podID="841ef677-3054-4950-8de6-29fac7a070eb" containerID="9eb1c5189975f5b0c253460b10317574a6dc79b1632401de7f22da95e79cab4e" exitCode=1
Jul 28 16:34:04.824313 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.824269    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"
Jul 28 16:34:04.824313 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.824277    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5" event={"ID":"841ef677-3054-4950-8de6-29fac7a070eb","Type":"ContainerDied","Data":"9eb1c5189975f5b0c253460b10317574a6dc79b1632401de7f22da95e79cab4e"}
Jul 28 16:34:04.824380 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.824334    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5" event={"ID":"841ef677-3054-4950-8de6-29fac7a070eb","Type":"ContainerDied","Data":"eb6fc4abdd8a261ac754526973c692ddcd964139120f8682816b89eeb1e8e38a"}
Jul 28 16:34:04.824380 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.824362    2581 scope.go:117] "RemoveContainer" containerID="9eb1c5189975f5b0c253460b10317574a6dc79b1632401de7f22da95e79cab4e"
Jul 28 16:34:04.836715 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.836693    2581 scope.go:117] "RemoveContainer" containerID="f82ea09a98114f49f88d53d08d7919826fabf4bddc74d2ad7f215ff53c7c88eb"
Jul 28 16:34:04.843801 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.843784    2581 scope.go:117] "RemoveContainer" containerID="9eb1c5189975f5b0c253460b10317574a6dc79b1632401de7f22da95e79cab4e"
Jul 28 16:34:04.844047 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:34:04.844029    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eb1c5189975f5b0c253460b10317574a6dc79b1632401de7f22da95e79cab4e\": container with ID starting with 9eb1c5189975f5b0c253460b10317574a6dc79b1632401de7f22da95e79cab4e not found: ID does not exist" containerID="9eb1c5189975f5b0c253460b10317574a6dc79b1632401de7f22da95e79cab4e"
Jul 28 16:34:04.844093 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.844055    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eb1c5189975f5b0c253460b10317574a6dc79b1632401de7f22da95e79cab4e"} err="failed to get container status \"9eb1c5189975f5b0c253460b10317574a6dc79b1632401de7f22da95e79cab4e\": rpc error: code = NotFound desc = could not find container \"9eb1c5189975f5b0c253460b10317574a6dc79b1632401de7f22da95e79cab4e\": container with ID starting with 9eb1c5189975f5b0c253460b10317574a6dc79b1632401de7f22da95e79cab4e not found: ID does not exist"
Jul 28 16:34:04.844093 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.844072    2581 scope.go:117] "RemoveContainer" containerID="f82ea09a98114f49f88d53d08d7919826fabf4bddc74d2ad7f215ff53c7c88eb"
Jul 28 16:34:04.844302 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:34:04.844284    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f82ea09a98114f49f88d53d08d7919826fabf4bddc74d2ad7f215ff53c7c88eb\": container with ID starting with f82ea09a98114f49f88d53d08d7919826fabf4bddc74d2ad7f215ff53c7c88eb not found: ID does not exist" containerID="f82ea09a98114f49f88d53d08d7919826fabf4bddc74d2ad7f215ff53c7c88eb"
Jul 28 16:34:04.844343 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.844308    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f82ea09a98114f49f88d53d08d7919826fabf4bddc74d2ad7f215ff53c7c88eb"} err="failed to get container status \"f82ea09a98114f49f88d53d08d7919826fabf4bddc74d2ad7f215ff53c7c88eb\": rpc error: code = NotFound desc = could not find container \"f82ea09a98114f49f88d53d08d7919826fabf4bddc74d2ad7f215ff53c7c88eb\": container with ID starting with f82ea09a98114f49f88d53d08d7919826fabf4bddc74d2ad7f215ff53c7c88eb not found: ID does not exist"
Jul 28 16:34:04.874878 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.874843    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"]
Jul 28 16:34:04.879904 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:04.879880    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-136677-predictor-84ccc7d97b-x9gf5"]
Jul 28 16:34:05.349866 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.349814    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"]
Jul 28 16:34:05.350125 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.350111    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="841ef677-3054-4950-8de6-29fac7a070eb" containerName="storage-initializer"
Jul 28 16:34:05.350176 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.350127    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="841ef677-3054-4950-8de6-29fac7a070eb" containerName="storage-initializer"
Jul 28 16:34:05.350176 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.350144    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="841ef677-3054-4950-8de6-29fac7a070eb" containerName="storage-initializer"
Jul 28 16:34:05.350176 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.350150    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="841ef677-3054-4950-8de6-29fac7a070eb" containerName="storage-initializer"
Jul 28 16:34:05.350272 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.350189    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="841ef677-3054-4950-8de6-29fac7a070eb" containerName="storage-initializer"
Jul 28 16:34:05.350302 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.350275    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="841ef677-3054-4950-8de6-29fac7a070eb" containerName="storage-initializer"
Jul 28 16:34:05.354701 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.354681    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:05.356740 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.356721    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-bad41-predictor-serving-cert\""
Jul 28 16:34:05.356854 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.356734    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:34:05.356854 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.356802    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:34:05.357064 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.357044    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:34:05.357123 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.357051    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-bad41-kube-rbac-proxy-sar-config\""
Jul 28 16:34:05.357123 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.357098    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-m9t97\""
Jul 28 16:34:05.357225 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.357184    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:34:05.363672 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.363639    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"]
Jul 28 16:34:05.518999 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.518959    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80a72382-70ba-421a-8569-aa48af26ce33-proxy-tls\") pod \"raw-sklearn-bad41-predictor-85c45bccd4-c4jpq\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") " pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:05.519196 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.519004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80a72382-70ba-421a-8569-aa48af26ce33-cabundle-cert\") pod \"raw-sklearn-bad41-predictor-85c45bccd4-c4jpq\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") " pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:05.519196 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.519098    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqftp\" (UniqueName: \"kubernetes.io/projected/80a72382-70ba-421a-8569-aa48af26ce33-kube-api-access-dqftp\") pod \"raw-sklearn-bad41-predictor-85c45bccd4-c4jpq\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") " pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:05.519285 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.519192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80a72382-70ba-421a-8569-aa48af26ce33-kserve-provision-location\") pod \"raw-sklearn-bad41-predictor-85c45bccd4-c4jpq\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") " pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:05.519285 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.519245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80a72382-70ba-421a-8569-aa48af26ce33-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-bad41-predictor-85c45bccd4-c4jpq\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") " pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:05.620365 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.620278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80a72382-70ba-421a-8569-aa48af26ce33-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-bad41-predictor-85c45bccd4-c4jpq\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") " pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:05.620365 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.620322    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80a72382-70ba-421a-8569-aa48af26ce33-proxy-tls\") pod \"raw-sklearn-bad41-predictor-85c45bccd4-c4jpq\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") " pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:05.620562 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.620432    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80a72382-70ba-421a-8569-aa48af26ce33-cabundle-cert\") pod \"raw-sklearn-bad41-predictor-85c45bccd4-c4jpq\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") " pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:05.620562 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.620468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqftp\" (UniqueName: \"kubernetes.io/projected/80a72382-70ba-421a-8569-aa48af26ce33-kube-api-access-dqftp\") pod \"raw-sklearn-bad41-predictor-85c45bccd4-c4jpq\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") " pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:05.620562 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.620522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80a72382-70ba-421a-8569-aa48af26ce33-kserve-provision-location\") pod \"raw-sklearn-bad41-predictor-85c45bccd4-c4jpq\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") " pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:05.620872 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.620850    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80a72382-70ba-421a-8569-aa48af26ce33-kserve-provision-location\") pod \"raw-sklearn-bad41-predictor-85c45bccd4-c4jpq\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") " pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:05.621087 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.621068    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80a72382-70ba-421a-8569-aa48af26ce33-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-bad41-predictor-85c45bccd4-c4jpq\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") " pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:05.621163 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.621093    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80a72382-70ba-421a-8569-aa48af26ce33-cabundle-cert\") pod \"raw-sklearn-bad41-predictor-85c45bccd4-c4jpq\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") " pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:05.622879 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.622854    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80a72382-70ba-421a-8569-aa48af26ce33-proxy-tls\") pod \"raw-sklearn-bad41-predictor-85c45bccd4-c4jpq\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") " pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:05.629699 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.629671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqftp\" (UniqueName: \"kubernetes.io/projected/80a72382-70ba-421a-8569-aa48af26ce33-kube-api-access-dqftp\") pod \"raw-sklearn-bad41-predictor-85c45bccd4-c4jpq\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") " pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:05.665584 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.665540    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:05.795367 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.795337    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"]
Jul 28 16:34:05.798657 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:34:05.798625    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a72382_70ba_421a_8569_aa48af26ce33.slice/crio-ca7728f08dd173a9f6d7fc26fea86d3d045ef8e5e67c45e961e726235f14f6da WatchSource:0}: Error finding container ca7728f08dd173a9f6d7fc26fea86d3d045ef8e5e67c45e961e726235f14f6da: Status 404 returned error can't find the container with id ca7728f08dd173a9f6d7fc26fea86d3d045ef8e5e67c45e961e726235f14f6da
Jul 28 16:34:05.829290 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.829255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" event={"ID":"80a72382-70ba-421a-8569-aa48af26ce33","Type":"ContainerStarted","Data":"ca7728f08dd173a9f6d7fc26fea86d3d045ef8e5e67c45e961e726235f14f6da"}
Jul 28 16:34:05.982955 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:05.982914    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="841ef677-3054-4950-8de6-29fac7a070eb" path="/var/lib/kubelet/pods/841ef677-3054-4950-8de6-29fac7a070eb/volumes"
Jul 28 16:34:06.833884 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:06.833849    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" event={"ID":"80a72382-70ba-421a-8569-aa48af26ce33","Type":"ContainerStarted","Data":"56e79dcdd122455990605c8072c6204855f3346dec24a003ba3f5c2aef34c456"}
Jul 28 16:34:09.845036 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:09.844939    2581 generic.go:356] "Generic (PLEG): container finished" podID="80a72382-70ba-421a-8569-aa48af26ce33" containerID="56e79dcdd122455990605c8072c6204855f3346dec24a003ba3f5c2aef34c456" exitCode=0
Jul 28 16:34:09.845036 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:09.844987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" event={"ID":"80a72382-70ba-421a-8569-aa48af26ce33","Type":"ContainerDied","Data":"56e79dcdd122455990605c8072c6204855f3346dec24a003ba3f5c2aef34c456"}
Jul 28 16:34:10.850482 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:10.850440    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" event={"ID":"80a72382-70ba-421a-8569-aa48af26ce33","Type":"ContainerStarted","Data":"7611c59d182a9722c35dbc8ff60ef89c33a87dd05a8a2b1067b7683416985cdb"}
Jul 28 16:34:10.850937 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:10.850520    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" event={"ID":"80a72382-70ba-421a-8569-aa48af26ce33","Type":"ContainerStarted","Data":"40b1cb6db40c151a58a6bfe888c7c0cfb9d06ac484ab7b2aec42d6d3d48b43df"}
Jul 28 16:34:10.850937 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:10.850743    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:10.875335 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:10.875285    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" podStartSLOduration=2.18727393 podStartE2EDuration="5.87527049s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.68799656s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:34:10.872424788 +0000 UTC m=+1937.430981351" watchObservedRunningTime="2026-07-28 16:34:10.87527049 +0000 UTC m=+1937.433827031"
Jul 28 16:34:11.854037 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:11.853988    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:11.855398 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:11.855373    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 16:34:12.857929 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:12.857882    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 16:34:17.862240 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:17.862211    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:34:17.862699 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:17.862673    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 16:34:27.863141 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:27.863094    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 16:34:37.862698 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:37.862653    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 16:34:47.863006 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:47.862916    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 16:34:57.862873 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:34:57.862830    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 16:35:07.863066 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:07.863006    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 16:35:17.862786 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:17.862745    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 16:35:27.864079 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:27.864036    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:35:35.448811 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:35.448777    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"]
Jul 28 16:35:35.449188 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:35.449038    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kserve-container" containerID="cri-o://40b1cb6db40c151a58a6bfe888c7c0cfb9d06ac484ab7b2aec42d6d3d48b43df" gracePeriod=30
Jul 28 16:35:35.449188 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:35.449083    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kube-rbac-proxy" containerID="cri-o://7611c59d182a9722c35dbc8ff60ef89c33a87dd05a8a2b1067b7683416985cdb" gracePeriod=30
Jul 28 16:35:36.156260 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:36.156226    2581 generic.go:356] "Generic (PLEG): container finished" podID="80a72382-70ba-421a-8569-aa48af26ce33" containerID="7611c59d182a9722c35dbc8ff60ef89c33a87dd05a8a2b1067b7683416985cdb" exitCode=2
Jul 28 16:35:36.156438 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:36.156305    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" event={"ID":"80a72382-70ba-421a-8569-aa48af26ce33","Type":"ContainerDied","Data":"7611c59d182a9722c35dbc8ff60ef89c33a87dd05a8a2b1067b7683416985cdb"}
Jul 28 16:35:37.858292 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:37.858250    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 28 16:35:37.863585 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:37.863566    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 16:35:39.996584 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:39.996561    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:35:40.041438 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.041408    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80a72382-70ba-421a-8569-aa48af26ce33-kube-rbac-proxy-sar-config\") pod \"80a72382-70ba-421a-8569-aa48af26ce33\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") "
Jul 28 16:35:40.041598 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.041445    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80a72382-70ba-421a-8569-aa48af26ce33-proxy-tls\") pod \"80a72382-70ba-421a-8569-aa48af26ce33\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") "
Jul 28 16:35:40.041598 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.041475    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80a72382-70ba-421a-8569-aa48af26ce33-kserve-provision-location\") pod \"80a72382-70ba-421a-8569-aa48af26ce33\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") "
Jul 28 16:35:40.041980 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.041783    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a72382-70ba-421a-8569-aa48af26ce33-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "80a72382-70ba-421a-8569-aa48af26ce33" (UID: "80a72382-70ba-421a-8569-aa48af26ce33"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:35:40.041980 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.041828    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a72382-70ba-421a-8569-aa48af26ce33-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "80a72382-70ba-421a-8569-aa48af26ce33" (UID: "80a72382-70ba-421a-8569-aa48af26ce33"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:35:40.043500 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.043471    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80a72382-70ba-421a-8569-aa48af26ce33-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "80a72382-70ba-421a-8569-aa48af26ce33" (UID: "80a72382-70ba-421a-8569-aa48af26ce33"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:35:40.142628 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.142599    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80a72382-70ba-421a-8569-aa48af26ce33-cabundle-cert\") pod \"80a72382-70ba-421a-8569-aa48af26ce33\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") "
Jul 28 16:35:40.142768 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.142638    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqftp\" (UniqueName: \"kubernetes.io/projected/80a72382-70ba-421a-8569-aa48af26ce33-kube-api-access-dqftp\") pod \"80a72382-70ba-421a-8569-aa48af26ce33\" (UID: \"80a72382-70ba-421a-8569-aa48af26ce33\") "
Jul 28 16:35:40.142768 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.142755    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80a72382-70ba-421a-8569-aa48af26ce33-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:35:40.142768 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.142766    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80a72382-70ba-421a-8569-aa48af26ce33-proxy-tls\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:35:40.142870 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.142774    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80a72382-70ba-421a-8569-aa48af26ce33-kserve-provision-location\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:35:40.143056 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.143000    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a72382-70ba-421a-8569-aa48af26ce33-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "80a72382-70ba-421a-8569-aa48af26ce33" (UID: "80a72382-70ba-421a-8569-aa48af26ce33"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:35:40.144705 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.144685    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a72382-70ba-421a-8569-aa48af26ce33-kube-api-access-dqftp" (OuterVolumeSpecName: "kube-api-access-dqftp") pod "80a72382-70ba-421a-8569-aa48af26ce33" (UID: "80a72382-70ba-421a-8569-aa48af26ce33"). InnerVolumeSpecName "kube-api-access-dqftp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:35:40.171562 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.171538    2581 generic.go:356] "Generic (PLEG): container finished" podID="80a72382-70ba-421a-8569-aa48af26ce33" containerID="40b1cb6db40c151a58a6bfe888c7c0cfb9d06ac484ab7b2aec42d6d3d48b43df" exitCode=0
Jul 28 16:35:40.171678 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.171586    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" event={"ID":"80a72382-70ba-421a-8569-aa48af26ce33","Type":"ContainerDied","Data":"40b1cb6db40c151a58a6bfe888c7c0cfb9d06ac484ab7b2aec42d6d3d48b43df"}
Jul 28 16:35:40.171678 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.171595    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"
Jul 28 16:35:40.171678 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.171634    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq" event={"ID":"80a72382-70ba-421a-8569-aa48af26ce33","Type":"ContainerDied","Data":"ca7728f08dd173a9f6d7fc26fea86d3d045ef8e5e67c45e961e726235f14f6da"}
Jul 28 16:35:40.171678 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.171660    2581 scope.go:117] "RemoveContainer" containerID="7611c59d182a9722c35dbc8ff60ef89c33a87dd05a8a2b1067b7683416985cdb"
Jul 28 16:35:40.179481 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.179265    2581 scope.go:117] "RemoveContainer" containerID="40b1cb6db40c151a58a6bfe888c7c0cfb9d06ac484ab7b2aec42d6d3d48b43df"
Jul 28 16:35:40.186185 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.186168    2581 scope.go:117] "RemoveContainer" containerID="56e79dcdd122455990605c8072c6204855f3346dec24a003ba3f5c2aef34c456"
Jul 28 16:35:40.192660 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.192645    2581 scope.go:117] "RemoveContainer" containerID="7611c59d182a9722c35dbc8ff60ef89c33a87dd05a8a2b1067b7683416985cdb"
Jul 28 16:35:40.192905 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:35:40.192886    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7611c59d182a9722c35dbc8ff60ef89c33a87dd05a8a2b1067b7683416985cdb\": container with ID starting with 7611c59d182a9722c35dbc8ff60ef89c33a87dd05a8a2b1067b7683416985cdb not found: ID does not exist" containerID="7611c59d182a9722c35dbc8ff60ef89c33a87dd05a8a2b1067b7683416985cdb"
Jul 28 16:35:40.192951 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.192911    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7611c59d182a9722c35dbc8ff60ef89c33a87dd05a8a2b1067b7683416985cdb"} err="failed to get container status \"7611c59d182a9722c35dbc8ff60ef89c33a87dd05a8a2b1067b7683416985cdb\": rpc error: code = NotFound desc = could not find container \"7611c59d182a9722c35dbc8ff60ef89c33a87dd05a8a2b1067b7683416985cdb\": container with ID starting with 7611c59d182a9722c35dbc8ff60ef89c33a87dd05a8a2b1067b7683416985cdb not found: ID does not exist"
Jul 28 16:35:40.192951 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.192927    2581 scope.go:117] "RemoveContainer" containerID="40b1cb6db40c151a58a6bfe888c7c0cfb9d06ac484ab7b2aec42d6d3d48b43df"
Jul 28 16:35:40.193154 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:35:40.193135    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40b1cb6db40c151a58a6bfe888c7c0cfb9d06ac484ab7b2aec42d6d3d48b43df\": container with ID starting with 40b1cb6db40c151a58a6bfe888c7c0cfb9d06ac484ab7b2aec42d6d3d48b43df not found: ID does not exist" containerID="40b1cb6db40c151a58a6bfe888c7c0cfb9d06ac484ab7b2aec42d6d3d48b43df"
Jul 28 16:35:40.193229 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.193162    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40b1cb6db40c151a58a6bfe888c7c0cfb9d06ac484ab7b2aec42d6d3d48b43df"} err="failed to get container status \"40b1cb6db40c151a58a6bfe888c7c0cfb9d06ac484ab7b2aec42d6d3d48b43df\": rpc error: code = NotFound desc = could not find container \"40b1cb6db40c151a58a6bfe888c7c0cfb9d06ac484ab7b2aec42d6d3d48b43df\": container with ID starting with 40b1cb6db40c151a58a6bfe888c7c0cfb9d06ac484ab7b2aec42d6d3d48b43df not found: ID does not exist"
Jul 28 16:35:40.193229 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.193180    2581 scope.go:117] "RemoveContainer" containerID="56e79dcdd122455990605c8072c6204855f3346dec24a003ba3f5c2aef34c456"
Jul 28 16:35:40.193479 ip-10-0-131-181 kubenswrapper[2581]: E0728 16:35:40.193455    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e79dcdd122455990605c8072c6204855f3346dec24a003ba3f5c2aef34c456\": container with ID starting with 56e79dcdd122455990605c8072c6204855f3346dec24a003ba3f5c2aef34c456 not found: ID does not exist" containerID="56e79dcdd122455990605c8072c6204855f3346dec24a003ba3f5c2aef34c456"
Jul 28 16:35:40.193559 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.193488    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e79dcdd122455990605c8072c6204855f3346dec24a003ba3f5c2aef34c456"} err="failed to get container status \"56e79dcdd122455990605c8072c6204855f3346dec24a003ba3f5c2aef34c456\": rpc error: code = NotFound desc = could not find container \"56e79dcdd122455990605c8072c6204855f3346dec24a003ba3f5c2aef34c456\": container with ID starting with 56e79dcdd122455990605c8072c6204855f3346dec24a003ba3f5c2aef34c456 not found: ID does not exist"
Jul 28 16:35:40.194871 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.194851    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"]
Jul 28 16:35:40.199501 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.199471    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-bad41-predictor-85c45bccd4-c4jpq"]
Jul 28 16:35:40.243497 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.243469    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80a72382-70ba-421a-8569-aa48af26ce33-cabundle-cert\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:35:40.243497 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:40.243496    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqftp\" (UniqueName: \"kubernetes.io/projected/80a72382-70ba-421a-8569-aa48af26ce33-kube-api-access-dqftp\") on node \"ip-10-0-131-181.ec2.internal\" DevicePath \"\""
Jul 28 16:35:41.981836 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:35:41.981804    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80a72382-70ba-421a-8569-aa48af26ce33" path="/var/lib/kubelet/pods/80a72382-70ba-421a-8569-aa48af26ce33/volumes"
Jul 28 16:37:50.268221 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:50.268145    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zf8vz_0f92d8f5-b089-4ccf-a2c2-524a0eb11ef4/global-pull-secret-syncer/0.log"
Jul 28 16:37:50.367772 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:50.367739    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r4p2b_201d9cf5-adcd-49dd-be6f-f8e20cedce09/konnectivity-agent/0.log"
Jul 28 16:37:50.435434 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:50.435405    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-181.ec2.internal_e97bc9874712512bf8be887ba1ad9d3d/haproxy/0.log"
Jul 28 16:37:55.540551 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:55.540519    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5cvfq_dc3defc1-78da-434a-a933-cba987e0d5c0/node-exporter/0.log"
Jul 28 16:37:55.573111 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:55.573086    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5cvfq_dc3defc1-78da-434a-a933-cba987e0d5c0/kube-rbac-proxy/0.log"
Jul 28 16:37:55.602749 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:55.602728    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5cvfq_dc3defc1-78da-434a-a933-cba987e0d5c0/init-textfile/0.log"
Jul 28 16:37:57.847066 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.847031    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"]
Jul 28 16:37:57.847445 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.847291    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kube-rbac-proxy"
Jul 28 16:37:57.847445 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.847303    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kube-rbac-proxy"
Jul 28 16:37:57.847445 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.847312    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kserve-container"
Jul 28 16:37:57.847445 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.847320    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kserve-container"
Jul 28 16:37:57.847445 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.847336    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="storage-initializer"
Jul 28 16:37:57.847445 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.847342    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="storage-initializer"
Jul 28 16:37:57.847445 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.847390    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kube-rbac-proxy"
Jul 28 16:37:57.847445 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.847399    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="80a72382-70ba-421a-8569-aa48af26ce33" containerName="kserve-container"
Jul 28 16:37:57.850252 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.850230    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:37:57.852323 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.852301    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j7d85\"/\"default-dockercfg-zs968\""
Jul 28 16:37:57.852582 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.852564    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j7d85\"/\"openshift-service-ca.crt\""
Jul 28 16:37:57.853280 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.853256    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j7d85\"/\"kube-root-ca.crt\""
Jul 28 16:37:57.864469 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.864450    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"]
Jul 28 16:37:57.992210 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.992181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/43a57779-c0b1-4104-af56-9653a4efcdde-podres\") pod \"perf-node-gather-daemonset-pzbx8\" (UID: \"43a57779-c0b1-4104-af56-9653a4efcdde\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:37:57.992369 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.992217    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43a57779-c0b1-4104-af56-9653a4efcdde-lib-modules\") pod \"perf-node-gather-daemonset-pzbx8\" (UID: \"43a57779-c0b1-4104-af56-9653a4efcdde\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:37:57.992369 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.992274    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43a57779-c0b1-4104-af56-9653a4efcdde-sys\") pod \"perf-node-gather-daemonset-pzbx8\" (UID: \"43a57779-c0b1-4104-af56-9653a4efcdde\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:37:57.992369 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.992293    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bgl2\" (UniqueName: \"kubernetes.io/projected/43a57779-c0b1-4104-af56-9653a4efcdde-kube-api-access-2bgl2\") pod \"perf-node-gather-daemonset-pzbx8\" (UID: \"43a57779-c0b1-4104-af56-9653a4efcdde\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:37:57.992369 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:57.992309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/43a57779-c0b1-4104-af56-9653a4efcdde-proc\") pod \"perf-node-gather-daemonset-pzbx8\" (UID: \"43a57779-c0b1-4104-af56-9653a4efcdde\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:37:58.093063 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:58.093034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/43a57779-c0b1-4104-af56-9653a4efcdde-podres\") pod \"perf-node-gather-daemonset-pzbx8\" (UID: \"43a57779-c0b1-4104-af56-9653a4efcdde\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:37:58.093147 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:58.093072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43a57779-c0b1-4104-af56-9653a4efcdde-lib-modules\") pod \"perf-node-gather-daemonset-pzbx8\" (UID: \"43a57779-c0b1-4104-af56-9653a4efcdde\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:37:58.093147 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:58.093101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43a57779-c0b1-4104-af56-9653a4efcdde-sys\") pod \"perf-node-gather-daemonset-pzbx8\" (UID: \"43a57779-c0b1-4104-af56-9653a4efcdde\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:37:58.093227 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:58.093156    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43a57779-c0b1-4104-af56-9653a4efcdde-sys\") pod \"perf-node-gather-daemonset-pzbx8\" (UID: \"43a57779-c0b1-4104-af56-9653a4efcdde\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:37:58.093227 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:58.093180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bgl2\" (UniqueName: \"kubernetes.io/projected/43a57779-c0b1-4104-af56-9653a4efcdde-kube-api-access-2bgl2\") pod \"perf-node-gather-daemonset-pzbx8\" (UID: \"43a57779-c0b1-4104-af56-9653a4efcdde\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:37:58.093227 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:58.093196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/43a57779-c0b1-4104-af56-9653a4efcdde-proc\") pod \"perf-node-gather-daemonset-pzbx8\" (UID: \"43a57779-c0b1-4104-af56-9653a4efcdde\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:37:58.093227 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:58.093205    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/43a57779-c0b1-4104-af56-9653a4efcdde-podres\") pod \"perf-node-gather-daemonset-pzbx8\" (UID: \"43a57779-c0b1-4104-af56-9653a4efcdde\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:37:58.093361 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:58.093205    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43a57779-c0b1-4104-af56-9653a4efcdde-lib-modules\") pod \"perf-node-gather-daemonset-pzbx8\" (UID: \"43a57779-c0b1-4104-af56-9653a4efcdde\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:37:58.093361 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:58.093249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/43a57779-c0b1-4104-af56-9653a4efcdde-proc\") pod \"perf-node-gather-daemonset-pzbx8\" (UID: \"43a57779-c0b1-4104-af56-9653a4efcdde\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:37:58.103437 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:58.103384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bgl2\" (UniqueName: \"kubernetes.io/projected/43a57779-c0b1-4104-af56-9653a4efcdde-kube-api-access-2bgl2\") pod \"perf-node-gather-daemonset-pzbx8\" (UID: \"43a57779-c0b1-4104-af56-9653a4efcdde\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:37:58.159421 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:58.159395    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:37:58.289202 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:58.289172    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"]
Jul 28 16:37:58.291688 ip-10-0-131-181 kubenswrapper[2581]: W0728 16:37:58.291663    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod43a57779_c0b1_4104_af56_9653a4efcdde.slice/crio-038cd4ff108cf955a2c455f445d239e230725b10c9bf2331448d8813ed0b16ed WatchSource:0}: Error finding container 038cd4ff108cf955a2c455f445d239e230725b10c9bf2331448d8813ed0b16ed: Status 404 returned error can't find the container with id 038cd4ff108cf955a2c455f445d239e230725b10c9bf2331448d8813ed0b16ed
Jul 28 16:37:58.631111 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:58.631079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8" event={"ID":"43a57779-c0b1-4104-af56-9653a4efcdde","Type":"ContainerStarted","Data":"6a7bb677804a9bbb5d8d26259f6c91cf1abb42973ff1cb1c8520a28f01e63517"}
Jul 28 16:37:58.631295 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:58.631135    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8" event={"ID":"43a57779-c0b1-4104-af56-9653a4efcdde","Type":"ContainerStarted","Data":"038cd4ff108cf955a2c455f445d239e230725b10c9bf2331448d8813ed0b16ed"}
Jul 28 16:37:58.631295 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:58.631215    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:37:58.649582 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:37:58.649539    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8" podStartSLOduration=1.649525833 podStartE2EDuration="1.649525833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:37:58.648936944 +0000 UTC m=+2165.207493510" watchObservedRunningTime="2026-07-28 16:37:58.649525833 +0000 UTC m=+2165.208082375"
Jul 28 16:38:00.869463 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:00.869429    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2pd4c_64de3fa9-b45e-4425-8706-990230e0dcc9/dns/0.log"
Jul 28 16:38:00.903571 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:00.903547    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2pd4c_64de3fa9-b45e-4425-8706-990230e0dcc9/kube-rbac-proxy/0.log"
Jul 28 16:38:01.024876 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:01.024849    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9b4sm_441a3269-baa8-4575-99a6-007b21792801/dns-node-resolver/0.log"
Jul 28 16:38:01.730848 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:01.730812    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rwvsw_f4d7a739-0ea4-4d89-8fd4-f302b556a891/node-ca/0.log"
Jul 28 16:38:03.057694 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:03.057663    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v2tjl_4f762d9d-a0ac-49c4-8920-1feb6ea59c89/serve-healthcheck-canary/0.log"
Jul 28 16:38:03.685623 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:03.685591    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rq7b2_2a9f054b-0e6c-4f0b-a83e-235ce83c1a58/kube-rbac-proxy/0.log"
Jul 28 16:38:03.716422 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:03.716398    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rq7b2_2a9f054b-0e6c-4f0b-a83e-235ce83c1a58/exporter/0.log"
Jul 28 16:38:03.766993 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:03.766967    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rq7b2_2a9f054b-0e6c-4f0b-a83e-235ce83c1a58/extractor/0.log"
Jul 28 16:38:04.643819 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:04.643793    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-pzbx8"
Jul 28 16:38:06.222382 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:06.222350    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-7mjbz_3b8388b5-aff7-414a-904c-4f3f55ec4c08/server/0.log"
Jul 28 16:38:06.347278 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:06.347251    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-px782_42c4dbdb-2654-4200-a10a-15791b5c9efc/manager/0.log"
Jul 28 16:38:06.437304 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:06.437273    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-njxlk_3255b5f2-5bb3-4e3f-a698-f78e1ec82891/seaweedfs/0.log"
Jul 28 16:38:12.735964 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:12.735929    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-44lcg_ce86f1d2-93dd-4dde-9cda-f3e53fbd7642/kube-multus/0.log"
Jul 28 16:38:13.077259 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:13.077187    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nnbz6_a57f72c8-1133-4849-93f6-caa2d46580f4/kube-multus-additional-cni-plugins/0.log"
Jul 28 16:38:13.107039 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:13.106998    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nnbz6_a57f72c8-1133-4849-93f6-caa2d46580f4/egress-router-binary-copy/0.log"
Jul 28 16:38:13.136152 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:13.136123    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nnbz6_a57f72c8-1133-4849-93f6-caa2d46580f4/cni-plugins/0.log"
Jul 28 16:38:13.176697 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:13.176673    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nnbz6_a57f72c8-1133-4849-93f6-caa2d46580f4/bond-cni-plugin/0.log"
Jul 28 16:38:13.205204 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:13.205179    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nnbz6_a57f72c8-1133-4849-93f6-caa2d46580f4/routeoverride-cni/0.log"
Jul 28 16:38:13.234797 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:13.234773    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nnbz6_a57f72c8-1133-4849-93f6-caa2d46580f4/whereabouts-cni-bincopy/0.log"
Jul 28 16:38:13.271619 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:13.271599    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nnbz6_a57f72c8-1133-4849-93f6-caa2d46580f4/whereabouts-cni/0.log"
Jul 28 16:38:13.644185 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:13.644158    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zf8jb_c958899e-cac6-4e06-bf0a-e510c503c24d/network-metrics-daemon/0.log"
Jul 28 16:38:13.673501 ip-10-0-131-181 kubenswrapper[2581]: I0728 16:38:13.673475    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zf8jb_c958899e-cac6-4e06-bf0a-e510c503c24d/kube-rbac-proxy/0.log"